From 513fce5730a7b8a07b76f4813de1d4468b1fbdcd Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 28 Sep 2026 12:33:26 +0800 Subject: [PATCH 01/47] [feature](storage) Support native GCP IAM authentication ### What problem does this PR solve? Problem Summary: GCS access currently depends on HMAC credentials or unsigned requests. A deployment using Application Default Credentials or a Compute Engine service account cannot consistently use the same identity for object reads and writes, Iceberg metadata operations, storage vaults, and backend error-log downloads. Add native GCP OAuth authentication across FE, BE, and Cloud using `gs.credential_provider_type=DEFAULT` or `COMPUTE_ENGINE`, with optional `gs.impersonation_service_account`. GCP storage without explicit authentication defaults to Application Default Credentials. Public-bucket access requires explicit `ANONYMOUS`; existing HMAC configuration remains supported. Reject conflicting native, HMAC, and AWS-only authentication properties before creating clients. The implementation follows the current master architecture: - Keep credential resolution in the filesystem API, OAuth client construction in the GCS plugin, and Iceberg client integration in the Iceberg connector. Use the native Hadoop GCS connector for namespace operations and bearer-authenticated S3 XML clients for object operations. - Carry typed credentials through additive Thrift and Protobuf fields, Cloud object information, and the shared C++ S3 client factory. Preserve AWS authentication, client caching, and rate limiting. - Normalize GCS aliases for resources, serialize resource ALTER operations, and validate effective properties before publication. Preserve omitted vault credential fields during partial ALTER, support clearing impersonation explicitly, and reject anonymous storage vaults. Apply vault changes to candidate metadata so a failed update does not persist a partial rename. - Support `gs://` in EXPORT and OUTFILE while preserving HTTP paths. Generate backend GCS V4 error-log URLs through IAM `signBlob`, falling back to the local error-log path if signing fails. - Add unit coverage and IAM regression suites for catalogs, TVFs, load, export, outfile, backup/restore, resources, vaults, property validation, and anonymous access. Export tests read back the written data and isolate cleanup by run directory. Example native authentication properties: ```properties provider=GCP gs.credential_provider_type=DEFAULT gs.impersonation_service_account=target@project.iam.gserviceaccount.com ``` The impersonation property is optional. Use `COMPUTE_ENGINE` to select the attached VM identity explicitly, or `ANONYMOUS` for public object access. Compatibility and deployment: - Existing AWS/HMAC protocol fields and authentication paths remain available. Native GCP credentials require updated consumers; upgrade MetaService and the relevant FE/BE processes before enabling the feature during a rolling upgrade. - Package google-cloud-cpp OAuth libraries for BE/Cloud and the shaded Hadoop GCS connector for FE/Iceberg. - FE presigned uploads with native OAuth remain unsupported and fail explicitly. Backend error-log downloads use the dedicated GCS V4 signing path. - Native OAuth uses the existing configurable object-store endpoint, so this remains an outbound-request/SSRF surface. Catalog creation requires catalog CREATE privilege; vault creation requires global ADMIN privilege. Google token and signing operations use the credential library and Google IAM APIs. No remote code-loading path is introduced. ### Release note Add native GCP IAM authentication with Application Default Credentials, Compute Engine identities, and optional service-account impersonation for GCS-backed storage. GCP configurations without HMAC or an explicit authentication mode now use ADC; public access must select `ANONYMOUS`. Improve resource/vault credential updates and support GCS error-log signing. ### Validation Passed locally: - GCS filesystem reactor unit tests, including 88 filesystem API tests, 152 S3-base tests, and 36 GCS tests; zero failures, errors, or skips in these modules. - FE Core and Iceberg Checkstyle, plus the GCS reactor's style checks. - clang-format 16 checks for all 33 changed C++ files, BE build hygiene, connector import boundaries, and FE metadata-funnel checks. - Parsing of all 41 changed Java files; Protobuf descriptor generation and Thrift C++ generation for the changed definitions. - `git diff --check`. Not completed locally: - Full FE unit tests: the standard runner requires JDK 17; the available test runtime is JDK 20. - Iceberg reactor tests: generation stops because the installed Thrift compiler is 0.16.0 while master requires 0.24.0. The standalone Thrift syntax check above does not replace this build. - BE/Cloud compilation, unit tests, and clang-tidy: the installed thirdparty bundle lacks the required google-cloud-cpp libraries. - Live IAM regression suites: no configured Doris cluster or GCP test environment was available. ### Check List (For Author) - Test: - [x] Unit tests added; filesystem API/S3/GCS tests executed successfully. - [x] Regression tests added. - [ ] Full FE, Iceberg, BE/Cloud, and live IAM tests completed. - Behavior changed: - [x] Yes: native GCP authentication, ADC default, explicit anonymous mode, and validated partial credential updates as described above. - Does this need documentation? - [x] Yes: document credential properties, ADC deployment, impersonation permissions, explicit anonymous access, and upgrade requirements. --- be/CMakeLists.txt | 10 +- be/cmake/thirdparty.cmake | 3 + be/src/io/fs/gcs_signed_url_provider.cpp | 211 +++++++++ be/src/io/fs/gcs_signed_url_provider.h | 48 ++ be/src/io/fs/s3_file_system.cpp | 1 + be/src/runtime/runtime_state.cpp | 5 + be/src/util/obj_credential.cpp | 76 +++ be/src/util/s3_util.cpp | 110 ++++- be/src/util/s3_util.h | 9 +- .../common/auth/gcp/gcs_signed_url_test.cpp | 79 ++++ be/test/io/fs/s3_obj_storage_client_test.cpp | 11 + be/test/io/s3_client_factory_test.cpp | 296 ++++++++++++ cloud/CMakeLists.txt | 16 + cloud/cmake/thirdparty.cmake | 4 + cloud/src/common/CMakeLists.txt | 1 + cloud/src/common/auth/obj_credential.cpp | 239 ++++++++++ cloud/src/common/auth/obj_credential.h | 65 +++ .../meta-service/meta_service_resource.cpp | 104 ++++- cloud/src/recycler/s3_accessor.cpp | 18 +- cloud/src/recycler/s3_accessor.h | 3 + cloud/test/CMakeLists.txt | 4 + cloud/test/meta_service_test.cpp | 267 +++++++++++ cloud/test/obj_credential_test.cpp | 149 ++++++ cloud/test/s3_accessor_test.cpp | 39 ++ common/cpp/obj-client/auth/gcp/gcp_auth.cpp | 99 ++++ common/cpp/obj-client/auth/gcp/gcp_auth.h | 60 +++ .../cpp/obj-client/auth/gcp/gcp_s3_client.cpp | 51 ++ .../cpp/obj-client/auth/gcp/gcp_s3_client.h | 55 +++ .../auth/gcp/gcp_token_provider.cpp | 93 ++++ .../obj-client/auth/gcp/gcp_token_provider.h | 57 +++ .../obj-client/auth/gcp/gcs_signed_url.cpp | 194 ++++++++ .../cpp/obj-client/auth/gcp/gcs_signed_url.h | 62 +++ common/cpp/obj-client/auth/obj_credential.cpp | 134 ++++++ common/cpp/obj-client/auth/obj_credential.h | 70 +++ .../obj-client/auth/obj_credential_common.h | 88 ++++ .../obj-client/auth/obj_s3_client_factory.cpp | 54 +++ .../obj-client/auth/obj_s3_client_factory.h | 50 ++ .../cpp/obj-client/s3_obj_storage_client.cpp | 3 + common/cpp/obj-client/s3_obj_storage_client.h | 11 +- fe/be-java-extensions/hadoop-deps/pom.xml | 5 + fe/fe-connector/fe-connector-iceberg/pom.xml | 12 + .../iceberg/GcpS3FileIOAwsClientFactory.java | 74 +++ .../iceberg/IcebergCatalogFactory.java | 19 + .../iceberg/IcebergCatalogFactoryTest.java | 42 ++ fe/fe-core/pom.xml | 5 + .../org/apache/doris/analysis/BrokerDesc.java | 14 + .../apache/doris/analysis/OutFileClause.java | 5 + .../org/apache/doris/catalog/S3Resource.java | 145 ++++-- .../apache/doris/catalog/S3StorageVault.java | 6 +- .../apache/doris/catalog/StorageVaultMgr.java | 78 +++- .../doris/cloud/storage/ObjectInfo.java | 21 +- .../cloud/storage/ObjectInfoAdapter.java | 11 +- .../storage/auth/GcpCredentialAdapter.java | 71 +++ .../property/storage/auth/ObjCredential.java | 34 ++ .../storage/auth/ObjCredentialFactory.java | 66 +++ .../storage/CloudObjectStoreAdapter.java | 10 + .../datasource/storage/S3ResourceCompat.java | 1 + .../datasource/storage/S3ThriftAdapter.java | 19 + .../datasource/storage/StorageAdapter.java | 3 +- .../trees/plans/commands/ExportCommand.java | 6 +- .../analysis/StorageDescPersistTest.java | 21 + .../apache/doris/catalog/S3ResourceTest.java | 441 ++++++++++++++++++ .../cloud/catalog/S3StorageVaultTest.java | 253 ++++++++++ .../cloud/storage/ObjectInfoAdapterTest.java | 16 + .../nereids/trees/plans/OutFileTest.java | 104 +++++ .../plans/commands/ExportCommandTest.java | 116 ++++- .../trees/plans/commands/LoadCommandTest.java | 41 ++ .../ExternalFileTableValuedFunctionTest.java | 50 ++ .../doris/filesystem/auth/GcpCredential.java | 61 +++ .../auth/GcpCredentialProviderType.java | 35 ++ .../apache/doris/filesystem/auth/GcsAuth.java | 50 ++ .../filesystem/auth/GcsAuthResolver.java | 169 +++++++ .../filesystem/auth/GcsAuthResolverTest.java | 126 +++++ fe/fe-filesystem/fe-filesystem-gcs/pom.xml | 5 + .../apache/doris/filesystem/gcs/GcpAuth.java | 148 ++++++ .../gcs/GcsFileSystemProperties.java | 74 ++- .../filesystem/gcs/GcsFileSystemProvider.java | 14 +- .../doris/filesystem/gcs/GcsObjStorage.java | 58 +++ .../doris/filesystem/gcs/GcpAuthTest.java | 92 ++++ .../gcs/GcsFileSystemPropertiesTest.java | 54 ++- .../gcs/GcsFileSystemProviderTest.java | 11 + .../doris/filesystem/s3/S3CompatSignals.java | 6 +- .../filesystem/s3/S3FileSystemProperties.java | 4 + .../s3/S3FileSystemPropertiesTest.java | 6 + fe/pom.xml | 14 + gensrc/proto/cloud.proto | 20 + gensrc/thrift/AgentService.thrift | 18 + regression-test/conf/regression-conf.groovy | 22 + .../doris/regression/suite/Syncer.groovy | 16 +- .../util/ObjectStorageIamTestUtils.groovy | 193 ++++++++ .../test_alter_resource_with_role.groovy | 234 ++++++++++ .../test_alter_storage_vault_iam.groovy | 158 +++++++ .../test_backup_restore_iam.groovy | 101 ++++ .../test_catalog_instance_profile.groovy | 134 ++++++ .../test_catalog_with_role.groovy | 123 +++++ .../test_export_iam.groovy | 108 +++++ .../test_external_catalog_iam.groovy | 78 ++++ .../test_gcp_iam_property_validation.groovy | 131 ++++++ .../test_gcp_tvf_anonymous.groovy | 51 ++ ...test_gcp_vault_anonymous_validation.groovy | 42 ++ ...tables_catalog_credentials_provider.groovy | 110 +++++ .../test_load_iam.groovy | 106 +++++ .../test_resource_iam.groovy | 134 ++++++ .../test_select_into_outfile_iam.groovy | 80 ++++ .../test_storage_vault_iam.groovy | 80 ++++ .../test_tvf_anonymous.groovy | 77 +++ .../object_storage_iam_p0/test_tvf_iam.groovy | 66 +++ 107 files changed, 7413 insertions(+), 134 deletions(-) create mode 100644 be/src/io/fs/gcs_signed_url_provider.cpp create mode 100644 be/src/io/fs/gcs_signed_url_provider.h create mode 100644 be/src/util/obj_credential.cpp create mode 100644 be/test/common/auth/gcp/gcs_signed_url_test.cpp create mode 100644 cloud/src/common/auth/obj_credential.cpp create mode 100644 cloud/src/common/auth/obj_credential.h create mode 100644 cloud/test/obj_credential_test.cpp create mode 100644 common/cpp/obj-client/auth/gcp/gcp_auth.cpp create mode 100644 common/cpp/obj-client/auth/gcp/gcp_auth.h create mode 100644 common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp create mode 100644 common/cpp/obj-client/auth/gcp/gcp_s3_client.h create mode 100644 common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp create mode 100644 common/cpp/obj-client/auth/gcp/gcp_token_provider.h create mode 100644 common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp create mode 100644 common/cpp/obj-client/auth/gcp/gcs_signed_url.h create mode 100644 common/cpp/obj-client/auth/obj_credential.cpp create mode 100644 common/cpp/obj-client/auth/obj_credential.h create mode 100644 common/cpp/obj-client/auth/obj_credential_common.h create mode 100644 common/cpp/obj-client/auth/obj_s3_client_factory.cpp create mode 100644 common/cpp/obj-client/auth/obj_s3_client_factory.h create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java create mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java create mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsObjStorage.java create mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java create mode 100644 regression-test/framework/src/main/groovy/org/apache/doris/regression/util/ObjectStorageIamTestUtils.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_alter_storage_vault_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_backup_restore_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_export_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_gcp_iam_property_validation.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_gcp_tvf_anonymous.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_gcp_vault_anonymous_validation.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_load_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_resource_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_select_into_outfile_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_storage_vault_iam.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_tvf_anonymous.groovy create mode 100644 regression-test/suites/object_storage_iam_p0/test_tvf_iam.groovy diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt index 7f8b577814d1f8..1e430b31da6f58 100644 --- a/be/CMakeLists.txt +++ b/be/CMakeLists.txt @@ -674,7 +674,7 @@ set(DORIS_LINK_LIBS ) set(absl_DIR ${THIRDPARTY_DIR}/lib/cmake/absl) -find_package(absl) +find_package(absl CONFIG REQUIRED PATHS ${absl_DIR} NO_DEFAULT_PATH) # COMMON_THIRDPARTY are thirdparty dependencies that can run on all platform # When adding new dependencies, If you don’t know if it can run on all platforms, @@ -704,6 +704,14 @@ add_definitions(-DUSE_DORIS_HADOOP_HDFS) if (absl_FOUND) set(COMMON_THIRDPARTY ${COMMON_THIRDPARTY} + # Public dependencies of the statically linked google-cloud-cpp common library. + absl::base + absl::memory + absl::optional + absl::span + absl::str_format + absl::time + absl::variant absl::cord absl::cord_internal absl::cordz_functions diff --git a/be/cmake/thirdparty.cmake b/be/cmake/thirdparty.cmake index dcb7df7343b8b5..562bef86095c1a 100644 --- a/be/cmake/thirdparty.cmake +++ b/be/cmake/thirdparty.cmake @@ -69,6 +69,9 @@ add_thirdparty(lz4) add_thirdparty(lzo2) add_thirdparty(thrift) add_thirdparty(thriftnb) +add_thirdparty(google_cloud_cpp_oauth2) +add_thirdparty(google_cloud_cpp_rest_internal) +add_thirdparty(google_cloud_cpp_common) add_thirdparty(crc32c) add_thirdparty(libevent_core LIBNAME "lib/libevent_core.a") diff --git a/be/src/io/fs/gcs_signed_url_provider.cpp b/be/src/io/fs/gcs_signed_url_provider.cpp new file mode 100644 index 00000000000000..d69e0fa46e5bf4 --- /dev/null +++ b/be/src/io/fs/gcs_signed_url_provider.cpp @@ -0,0 +1,211 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "io/fs/gcs_signed_url_provider.h" + +#include +#include + +#include +#include +#include +#include +#include + +#include "cpp/obj-client/auth/gcp/gcp_token_provider.h" +#include "cpp/obj-client/auth/gcp/gcs_signed_url.h" +#include "service/http/http_client.h" +#include "util/string_util.h" +#include "util/url_coding.h" + +namespace doris::io { +namespace { + +constexpr std::string_view IAM_CREDENTIALS_ENDPOINT = + "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/"; +constexpr std::string_view DEFAULT_METADATA_HOST = "metadata.google.internal"; +constexpr std::string_view METADATA_SERVICE_ACCOUNT_EMAIL_PATH = + "/computeMetadata/v1/instance/service-accounts/default/email"; +constexpr int64_t METADATA_REQUEST_TIMEOUT_MS = 1000; + +std::string iam_error_message(const rapidjson::Document& document) { + if (!document.IsObject() || !document.HasMember("error") || !document["error"].IsObject()) { + return "unparseable error response"; + } + const auto& error = document["error"]; + std::string message; + if (error.HasMember("status") && error["status"].IsString()) { + message = error["status"].GetString(); + } + if (error.HasMember("message") && error["message"].IsString()) { + if (!message.empty()) { + message.append(": "); + } + message.append(error["message"].GetString()); + } + if (message.empty()) { + return "error response did not contain status or message"; + } + constexpr size_t MAX_ERROR_MESSAGE_SIZE = 1024; + if (message.size() > MAX_ERROR_MESSAGE_SIZE) { + message.resize(MAX_ERROR_MESSAGE_SIZE); + message.append("..."); + } + return message; +} + +bool is_unreserved(unsigned char c) { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || + c == '_' || c == '.' || c == '~'; +} + +std::string percent_encode(std::string_view value) { + constexpr char HEX[] = "0123456789ABCDEF"; + std::string encoded; + encoded.reserve(value.size()); + for (unsigned char c : value) { + if (is_unreserved(c)) { + encoded.push_back(static_cast(c)); + continue; + } + encoded.push_back('%'); + encoded.push_back(HEX[c >> 4]); + encoded.push_back(HEX[c & 0x0F]); + } + return encoded; +} + +Status call_iam_sign_blob(std::string_view access_token, std::string_view service_account, + std::string_view string_to_sign, int64_t request_timeout_ms, + std::string* signature) { + std::string encoded_payload; + base64_encode(std::string(string_to_sign), &encoded_payload); + std::string request_body = fmt::format(R"({{"payload":"{}"}})", encoded_payload); + std::string endpoint = + std::string(IAM_CREDENTIALS_ENDPOINT) + percent_encode(service_account) + ":signBlob"; + + HttpClient client; + // Keep the response body for non-2xx replies so IAM permission and + // service-account errors are actionable to operators. + RETURN_IF_ERROR(client.init(endpoint, false)); + client.set_authorization("Bearer " + std::string(access_token)); + client.set_content_type("application/json"); + client.set_timeout_ms(request_timeout_ms > 0 ? request_timeout_ms : 10000); + + std::string response; + RETURN_IF_ERROR(client.execute_post_request(request_body, &response)); + + rapidjson::Document document; + document.Parse(response.data(), response.size()); + const auto http_status = client.get_http_status(); + if (http_status < 200 || http_status >= 300) { + return Status::HttpError("IAM signBlob failed with HTTP {}: {}", http_status, + iam_error_message(document)); + } + if (document.HasParseError() || !document.IsObject() || !document.HasMember("signedBlob") || + !document["signedBlob"].IsString()) { + return Status::InternalError("IAM signBlob returned an invalid response"); + } + if (!base64_decode(document["signedBlob"].GetString(), signature)) { + return Status::InternalError("IAM signBlob returned an invalid base64 signature"); + } + return Status::OK(); +} + +Status fetch_metadata_service_account_email(int64_t request_timeout_ms, std::string* email) { + const char* configured_host = std::getenv("GCE_METADATA_HOST"); + std::string_view metadata_host = configured_host != nullptr && configured_host[0] != '\0' + ? configured_host + : DEFAULT_METADATA_HOST; + HttpClient client; + RETURN_IF_ERROR(client.init( + fmt::format("http://{}{}", metadata_host, METADATA_SERVICE_ACCOUNT_EMAIL_PATH))); + client.set_header("Metadata-Flavor", "Google"); + client.set_timeout_ms(request_timeout_ms > 0 + ? std::min(request_timeout_ms, METADATA_REQUEST_TIMEOUT_MS) + : METADATA_REQUEST_TIMEOUT_MS); + + std::string response; + RETURN_IF_ERROR(client.execute(&response)); + auto resolved_email = trim(response); + if (!is_valid_gcp_service_account_email(resolved_email)) { + return Status::InternalError( + "GCP metadata server returned an invalid service account email"); + } + email->assign(resolved_email); + return Status::OK(); +} + +} // namespace + +Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, + const GcpCredentialConfig& credential, + const std::shared_ptr& token_provider, + std::string* signed_url) { + if (signed_url == nullptr) { + return Status::InvalidArgument("signed_url output must not be null"); + } + signed_url->clear(); + + std::string signer_email; + if (!credential.impersonation_service_account.empty()) { + signer_email = credential.impersonation_service_account; + } else if (credential.provider_type == GcpCredentialProviderType::ComputeEngine) { + RETURN_IF_ERROR( + fetch_metadata_service_account_email(options.request_timeout_ms, &signer_email)); + } else { + return Status::InvalidArgument( + "GCS V4 signing with DEFAULT credentials requires " + "gs.impersonation_service_account; use COMPUTE_ENGINE to resolve the VM service " + "account from metadata"); + } + if (token_provider == nullptr) { + return Status::InternalError("GCS V4 signing token provider is not initialized"); + } + + try { + auto token = token_provider->get_token(); + if (!token.has_value()) { + return Status::InternalError("failed to obtain OAuth token for IAM signBlob"); + } + + auto result = build_gcs_v4_signed_url( + {.endpoint = options.endpoint, + .bucket = options.bucket, + .key = options.key, + .signer_email = signer_email, + .expiration_secs = options.expiration_secs}, + std::chrono::system_clock::now(), [&](std::string_view string_to_sign) { + std::string signature; + auto status = call_iam_sign_blob(*token, signer_email, string_to_sign, + options.request_timeout_ms, &signature); + if (!status.ok()) { + return GcsSignBlobResult {.error = status.to_string()}; + } + return GcsSignBlobResult {.signature = std::move(signature)}; + }); + if (!result.ok()) { + return Status::InternalError("failed to build GCS V4 signed URL: {}", result.error); + } + *signed_url = std::move(result.signed_url); + return Status::OK(); + } catch (const std::exception& e) { + return Status::InternalError("failed to initialize GCS URL signer: {}", e.what()); + } +} + +} // namespace doris::io diff --git a/be/src/io/fs/gcs_signed_url_provider.h b/be/src/io/fs/gcs_signed_url_provider.h new file mode 100644 index 00000000000000..771a247bf2d112 --- /dev/null +++ b/be/src/io/fs/gcs_signed_url_provider.h @@ -0,0 +1,48 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include "common/status.h" +#include "cpp/obj-client/auth/gcp/gcp_auth.h" + +namespace doris { +class GcpTokenProvider; +} + +namespace doris::io { + +struct GcsV4SignedUrlProviderOptions { + std::string endpoint; + std::string bucket; + std::string key; + int64_t expiration_secs = 0; + int64_t request_timeout_ms = 10000; +}; + +// Resolves the signing service account, obtains an OAuth token, and invokes +// IAM Credentials signBlob before delegating URL construction to common auth. +Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, + const GcpCredentialConfig& credential, + const std::shared_ptr& token_provider, + std::string* signed_url); + +} // namespace doris::io diff --git a/be/src/io/fs/s3_file_system.cpp b/be/src/io/fs/s3_file_system.cpp index d307335d15a517..4797b573961a77 100644 --- a/be/src/io/fs/s3_file_system.cpp +++ b/be/src/io/fs/s3_file_system.cpp @@ -98,6 +98,7 @@ Status ObjClientHolder::reset(const S3ClientConf& conf) { reset_conf.role_arn = conf.role_arn; reset_conf.external_id = conf.external_id; reset_conf.cred_provider_type = conf.cred_provider_type; + reset_conf.credential = conf.credential; // Compare full-field equality of the merged conf, not get_hash(): the hash is // an XOR of crc32s and distinct configurations can collide, which would skip a diff --git a/be/src/runtime/runtime_state.cpp b/be/src/runtime/runtime_state.cpp index 50740802417aeb..dfd45565b599e2 100644 --- a/be/src/runtime/runtime_state.cpp +++ b/be/src/runtime/runtime_state.cpp @@ -533,6 +533,11 @@ std::string RuntimeState::get_error_log_file_path() { auto presigned_url = s3_error_fs->generate_presigned_url(remote_error_log_file_path, EXPIRATION_SECONDS, config::use_public_endpoint_for_error_log); + if (presigned_url.empty()) { + LOG(WARNING) << "Fail to generate presigned URL for S3 error log; retain local path " + << local_error_log_file_path; + return local_error_log_file_path; + } std::lock_guard load_lock(_load_error_log_lock); _error_log_file_path = std::move(presigned_url); return _error_log_file_path; diff --git a/be/src/util/obj_credential.cpp b/be/src/util/obj_credential.cpp new file mode 100644 index 00000000000000..6308350c0e441f --- /dev/null +++ b/be/src/util/obj_credential.cpp @@ -0,0 +1,76 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/obj_credential.h" + +#include +#include + +namespace doris { +namespace { + +GcpCredentialConfig to_gcp_credential_config(const cloud::GcpCredentialPB& credential) { + GcpCredentialConfig config; + switch (credential.credential_provider_type()) { + case cloud::GcpCredentialPB::DEFAULT: + config.provider_type = GcpCredentialProviderType::Default; + break; + case cloud::GcpCredentialPB::COMPUTE_ENGINE: + config.provider_type = GcpCredentialProviderType::ComputeEngine; + break; + } + config.impersonation_service_account = credential.impersonation_service_account(); + return config; +} + +GcpCredentialConfig to_gcp_credential_config(const TGcpCredential& credential) { + GcpCredentialConfig config; + if (credential.__isset.credential_provider_type) { + switch (credential.credential_provider_type) { + case TGcpCredentialProviderType::DEFAULT: + config.provider_type = GcpCredentialProviderType::Default; + break; + case TGcpCredentialProviderType::COMPUTE_ENGINE: + config.provider_type = GcpCredentialProviderType::ComputeEngine; + break; + } + } + if (credential.__isset.impersonation_service_account) { + config.impersonation_service_account = credential.impersonation_service_account; + } + return config; +} + +} // namespace + +void convert_obj_credential(const cloud::ObjectStoreInfoPB& source, CredentialConfig* credential) { + if (!source.has_credential() || !source.credential().has_gcp_credential()) { + *credential = std::monostate {}; + return; + } + *credential = to_gcp_credential_config(source.credential().gcp_credential()); +} + +void convert_obj_credential(const TCredential& source, CredentialConfig* credential) { + if (!source.__isset.gcp_credential) { + *credential = std::monostate {}; + return; + } + *credential = to_gcp_credential_config(source.gcp_credential); +} + +} // namespace doris diff --git a/be/src/util/s3_util.cpp b/be/src/util/s3_util.cpp index d42bcae905531f..29c7b15ff3111a 100644 --- a/be/src/util/s3_util.cpp +++ b/be/src/util/s3_util.cpp @@ -51,12 +51,16 @@ #endif #include "cloud/config.h" #include "cpp/aws_logger.h" +#include "cpp/obj-client/auth/gcp/gcp_token_provider.h" +#include "cpp/obj-client/auth/obj_credential.h" +#include "cpp/obj-client/auth/obj_s3_client_factory.h" #include "cpp/obj-client/rate_limited_obj_storage_client.h" #include "cpp/obj-client/s3_obj_storage_client.h" #include "cpp/obj_retry_strategy.h" #include "cpp/sync_point.h" #include "cpp/util.h" #include "exec/scan/scanner_scheduler.h" +#include "io/fs/gcs_signed_url_provider.h" #include "runtime/exec_env.h" #include "util/s3_rate_limiter_manager.h" #include "util/s3_uri.h" @@ -64,6 +68,32 @@ namespace doris { namespace { +// Adapter between the IO storage-domain enum and the credential layer. Keep it +// local: neither layer should own or depend on the other's enum definition. +ObjCredentialProvider to_obj_credential_provider(io::ObjStorageProvider provider) { + switch (provider) { + case io::ObjStorageProvider::AWS: + return ObjCredentialProvider::Aws; + case io::ObjStorageProvider::AZURE: + return ObjCredentialProvider::Azure; + case io::ObjStorageProvider::BOS: + return ObjCredentialProvider::Bos; + case io::ObjStorageProvider::COS: + return ObjCredentialProvider::Cos; + case io::ObjStorageProvider::GCP: + return ObjCredentialProvider::Gcp; + case io::ObjStorageProvider::OBS: + return ObjCredentialProvider::Obs; + case io::ObjStorageProvider::OSS: + return ObjCredentialProvider::Oss; + case io::ObjStorageProvider::TOS: + return ObjCredentialProvider::Tos; + case io::ObjStorageProvider::UNKNOWN: + return ObjCredentialProvider::Unknown; + } + return ObjCredentialProvider::Unknown; +} + doris::Status is_s3_conf_valid(const S3ClientConf& conf) { if (conf.endpoint.empty()) { return Status::InvalidArgument("Invalid s3 conf, empty endpoint"); @@ -72,6 +102,20 @@ doris::Status is_s3_conf_valid(const S3ClientConf& conf) { return Status::InvalidArgument("Invalid s3 conf, empty region"); } + ObjCredentialValidationContext credential_context { + .provider = to_obj_credential_provider(conf.provider), + .ak = conf.ak, + .sk = conf.sk, + .token = conf.token, + .has_aws_role_arn = !conf.role_arn.empty(), + .has_aws_external_id = !conf.external_id.empty(), + .has_aws_credential_provider = conf.cred_provider_type != CredProviderType::Default, + }; + if (auto error = validate_obj_credential_config(conf.credential, credential_context); + error.has_value()) { + return Status::InvalidArgument(*error); + } + if (conf.role_arn.empty()) { // Allow anonymous access when both ak and sk are empty bool hasAk = !conf.ak.empty(); @@ -151,6 +195,7 @@ std::string build_azure_tls_debug_context(const std::string& selected_ca_file) { constexpr char USE_PATH_STYLE[] = "use_path_style"; constexpr char AZURE_PROVIDER_STRING[] = "AZURE"; +constexpr char GCP_PROVIDER_STRING[] = "GCP"; constexpr char S3_PROVIDER[] = "provider"; constexpr char S3_AK[] = "AWS_ACCESS_KEY"; constexpr char S3_SK[] = "AWS_SECRET_KEY"; @@ -389,17 +434,53 @@ Result> S3ClientFactory::_create_s3_client return ResultError(Status::InvalidArgument("failed to create AWS credential provider: {}", credentials.error)); } - std::shared_ptr new_client = std::make_shared( - std::move(credentials.provider), std::move(aws_config), - Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, - s3_conf.use_virtual_addressing); + auto new_client = make_s3_client( + s3_conf.credential, + S3ClientBuildContext { + .fallback_provider = std::move(credentials.provider), + .config = std::move(aws_config), + .payload_signing_policy = + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + .use_virtual_addressing = s3_conf.use_virtual_addressing, + .ca_cert_path = _ca_cert_file_path, + }); + std::shared_ptr gcs_signing_token_provider; + if (const auto* gcp_credential = std::get_if(&s3_conf.credential)) { + // signBlob is authorized by the source identity, not by the optional + // impersonated target identity. Keep one refreshable cloud-platform + // token provider for the lifetime of the S3 client. + auto signing_caller_credential = *gcp_credential; + signing_caller_credential.impersonation_service_account.clear(); + gcs_signing_token_provider = std::make_shared( + signing_caller_credential, _ca_cert_file_path, GcpTokenScope::CloudPlatform); + } + + S3ObjStorageClient::SignedUrlGenerator signed_url_generator; + if (const auto* credential = std::get_if(&s3_conf.credential)) { + signed_url_generator = [conf = s3_conf, credential = *credential, + token_provider = std::move(gcs_signing_token_provider)]( + const ObjStoragePath& path, int64_t expiration_secs) { + std::string signed_url; + auto status = + io::generate_gcs_v4_signed_url({.endpoint = conf.endpoint, + .bucket = path.bucket, + .key = path.key, + .expiration_secs = expiration_secs, + .request_timeout_ms = conf.request_timeout_ms}, + credential, token_provider, &signed_url); + if (!status.ok()) { + LOG(WARNING) << "failed to generate GCS V4 signed URL: " << status; + return std::string {}; + } + return signed_url; + }; + } auto provider_client = std::make_shared( - std::move(new_client), ObjStorageEndpointInfo { - .endpoint = s3_conf.endpoint, - .ak = s3_conf.ak, - .sk = s3_conf.sk, - }); + std::move(new_client), + ObjStorageEndpointInfo { + .endpoint = s3_conf.endpoint, .ak = s3_conf.ak, .sk = s3_conf.sk}, + std::move(signed_url_generator)); LOG_INFO("create one s3 client with {}", s3_conf.to_string()); return provider_client; } @@ -446,6 +527,8 @@ Status S3ClientFactory::convert_properties_to_s3_conf( // S3 Provider properties should be case insensitive. if (0 == strcasecmp(it->second.c_str(), AZURE_PROVIDER_STRING)) { s3_conf->client_conf.provider = io::ObjStorageProvider::AZURE; + } else if (0 == strcasecmp(it->second.c_str(), GCP_PROVIDER_STRING)) { + s3_conf->client_conf.provider = io::ObjStorageProvider::GCP; } } @@ -478,6 +561,11 @@ Status S3ClientFactory::convert_properties_to_s3_conf( s3_conf->client_conf.cred_provider_type = cred_provider_type_from_string(it->second); } + if (auto error = parse_obj_credential_properties(properties, &s3_conf->client_conf.credential); + error.has_value()) { + return Status::InvalidArgument(*error); + } + if (auto st = is_s3_conf_valid(s3_conf->client_conf); !st.ok()) { return st; } @@ -537,6 +625,7 @@ S3Conf S3Conf::get_s3_conf(const cloud::ObjectStoreInfoPB& info) { if (info.has_cred_provider_type()) { ret.client_conf.cred_provider_type = cred_provider_type_from_pb(info.cred_provider_type()); } + convert_obj_credential(info, &ret.client_conf.credential); io::ObjStorageProvider type = io::ObjStorageProvider::AWS; switch (info.provider()) { @@ -598,6 +687,9 @@ S3Conf S3Conf::get_s3_conf(const TS3StorageParam& param) { ret.client_conf.cred_provider_type = cred_provider_type_from_thrift(param.cred_provider_type); } + if (param.__isset.credential) { + convert_obj_credential(param.credential, &ret.client_conf.credential); + } io::ObjStorageProvider type = io::ObjStorageProvider::AWS; switch (param.provider) { diff --git a/be/src/util/s3_util.h b/be/src/util/s3_util.h index b2ca49d4751cb9..6856d205ff5241 100644 --- a/be/src/util/s3_util.h +++ b/be/src/util/s3_util.h @@ -39,6 +39,7 @@ #include "core/string_ref.h" #include "cpp/aws_common.h" #include "cpp/obj-client/auth/aws_credential_factory.h" +#include "cpp/obj-client/auth/obj_credential.h" #include "cpp/obj-client/obj_storage_client.h" namespace Aws::S3 { @@ -74,6 +75,7 @@ struct S3ClientConf { CredProviderType cred_provider_type = CredProviderType::Default; std::string role_arn; std::string external_id; + CredentialConfig credential {}; // True when this client is bound to a Doris internal object storage bucket // (a storage vault in cloud mode). S3ClientFactory wraps such clients with the // shared rate limiter; external buckets (S3 load, TVF, external catalogs) are @@ -102,6 +104,7 @@ struct S3ClientConf { hash_code ^= static_cast(cred_provider_type); hash_code ^= crc32_hash(role_arn); hash_code ^= crc32_hash(external_id); + hash_combine_credential(&hash_code, credential); hash_code ^= is_internal_bucket; return hash_code; } @@ -110,10 +113,12 @@ struct S3ClientConf { return fmt::format( "(ak={}, token={}, endpoint={}, region={}, bucket={}, max_connections={}, " "request_timeout_ms={}, connect_timeout_ms={}, use_virtual_addressing={}, " - "cred_provider_type={},role_arn={}, external_id={}, is_internal_bucket={}", + "cred_provider_type={},role_arn={}, external_id={}, credential_type={}, " + "is_internal_bucket={}", hide_access_key(ak), token.empty() ? "" : "******", endpoint, region, bucket, max_connections, request_timeout_ms, connect_timeout_ms, use_virtual_addressing, - cred_provider_type, role_arn, external_id, is_internal_bucket); + cred_provider_type, role_arn, external_id, credential_type_name(credential), + is_internal_bucket); } }; diff --git a/be/test/common/auth/gcp/gcs_signed_url_test.cpp b/be/test/common/auth/gcp/gcs_signed_url_test.cpp new file mode 100644 index 00000000000000..f4a6292ae23b63 --- /dev/null +++ b/be/test/common/auth/gcp/gcs_signed_url_test.cpp @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/gcp/gcs_signed_url.h" + +#include + +#include +#include + +namespace doris { +namespace { + +TEST(GcsSignedUrlTest, BuildsV4UrlAndPassesCanonicalStringToSigner) { + GcsV4SignedUrlOptions options { + .endpoint = "storage.googleapis.com/", + .bucket = "test-bucket", + .key = "error_log/a b+%.csv", + .signer_email = "signer@my-project.iam.gserviceaccount.com", + .expiration_secs = 60, + }; + std::string captured_string_to_sign; + auto result = build_gcs_v4_signed_url( + options, std::chrono::system_clock::time_point {}, + [&](std::string_view string_to_sign) { + captured_string_to_sign = string_to_sign; + return GcsSignBlobResult {.signature = std::string("\x00\xab\xff", 3)}; + }); + + ASSERT_TRUE(result.ok()) << result.error; + EXPECT_EQ(result.signed_url, + "https://storage.googleapis.com/test-bucket/error_log/a%20b%2B%25.csv?" + "X-Goog-Algorithm=GOOG4-RSA-SHA256&" + "X-Goog-Credential=signer%40my-project.iam.gserviceaccount.com%2F19700101%2Fauto%" + "2Fstorage%2Fgoog4_request&X-Goog-Date=19700101T000000Z&X-Goog-Expires=60&" + "X-Goog-SignedHeaders=host&X-Goog-Signature=00abff"); + + EXPECT_EQ(captured_string_to_sign, + "GOOG4-RSA-SHA256\n19700101T000000Z\n" + "19700101/auto/storage/goog4_request\n" + "f741db2e699530357ae34622ae689c7ec43e10e418af0ff7f1487beb1f144ce5"); +} + +TEST(GcsSignedUrlTest, RejectsInvalidExpirationBeforeSigning) { + GcsV4SignedUrlOptions options { + .endpoint = "storage.googleapis.com", + .bucket = "test-bucket", + .key = "error_log/file", + .signer_email = "signer@my-project.iam.gserviceaccount.com", + .expiration_secs = 7 * 24 * 60 * 60 + 1, + }; + bool signer_called = false; + auto result = build_gcs_v4_signed_url(options, std::chrono::system_clock::time_point {}, + [&](std::string_view) { + signer_called = true; + return GcsSignBlobResult {}; + }); + + EXPECT_FALSE(result.ok()); + EXPECT_FALSE(signer_called); + EXPECT_TRUE(result.signed_url.empty()); +} + +} // namespace +} // namespace doris diff --git a/be/test/io/fs/s3_obj_storage_client_test.cpp b/be/test/io/fs/s3_obj_storage_client_test.cpp index 4bafbc4dde71f7..f43fd498de7a7a 100644 --- a/be/test/io/fs/s3_obj_storage_client_test.cpp +++ b/be/test/io/fs/s3_obj_storage_client_test.cpp @@ -25,6 +25,17 @@ namespace doris { +TEST(S3ObjStorageSignedUrlTest, DelegatesProviderSpecificSigning) { + S3ObjStorageClient client(nullptr, {}, [](const ObjStoragePath& path, int64_t expiration) { + EXPECT_EQ(path.bucket, "bucket"); + EXPECT_EQ(path.key, "nested/key"); + EXPECT_EQ(expiration, 300); + return std::string {"https://storage.googleapis.com/bucket/nested/key?signed"}; + }); + EXPECT_EQ(client.generate_presigned_url({.bucket = "bucket", .key = "nested/key"}, 300), + "https://storage.googleapis.com/bucket/nested/key?signed"); +} + class S3ObjStorageClientTest : public testing::Test { protected: static std::shared_ptr obj_storage_client; diff --git a/be/test/io/s3_client_factory_test.cpp b/be/test/io/s3_client_factory_test.cpp index 835b32bdaef7e5..c996cd4f105bb2 100644 --- a/be/test/io/s3_client_factory_test.cpp +++ b/be/test/io/s3_client_factory_test.cpp @@ -18,8 +18,12 @@ #include #include #include +#include #include +#include +#include #include +#include #include #include #include @@ -30,6 +34,7 @@ #include #include #include +#include #include #include @@ -38,8 +43,12 @@ #include "cpp/aws_common.h" #include "cpp/custom_aws_credentials_provider_chain.h" #include "cpp/obj-client/auth/aws_credential_factory.h" +#include "cpp/obj-client/auth/gcp/gcp_auth.h" +#include "cpp/obj-client/auth/gcp/gcp_s3_client.h" +#include "cpp/obj-client/rate_limited_obj_storage_client.h" #include "cpp/obj-client/s3_obj_storage_client.h" #include "cpp/sync_point.h" +#include "io/fs/err_utils.h" #include "io/fs/s3_file_system.h" #include "testutil/container_credentials_endpoint.h" #include "util/s3_rate_limiter_manager.h" @@ -126,6 +135,30 @@ class RateLimiterConfigGuard { size_t _bytes_limit; }; +S3ClientConf make_gcp_native_conf(std::string impersonation_service_account = {}) { + auto conf = make_factory_conf("storage.googleapis.com", true); + conf.provider = io::ObjStorageProvider::GCP; + conf.region = "us-east1"; + conf.cred_provider_type = CredProviderType::Default; + conf.credential = GcpCredentialConfig { + .provider_type = GcpCredentialProviderType::Default, + .impersonation_service_account = std::move(impersonation_service_account), + }; + return conf; +} + +class FixedTokenGcpS3Client final : public GcpS3Client { +public: + FixedTokenGcpS3Client() + : GcpS3Client(GcpCredentialConfig {}, "", Aws::Client::ClientConfiguration {}, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true) {} + + using GcpS3Client::BuildHttpRequest; + +protected: + std::optional fetch_token() const override { return "test-gcp-token"; } +}; + class SyncPointProcessingGuard { public: SyncPointProcessingGuard() : _was_enabled(SyncPoint::get_instance()->get_enable()) { @@ -282,6 +315,119 @@ TEST_F(S3ClientFactoryTest, RateLimitResponseDistinguishesBytesFromProviderThrot EXPECT_NE(response.status.msg.find("s3 get request exceeds bytes limit"), std::string::npos); } +TEST_F(S3ClientFactoryTest, ObjClientHolderResetRefreshesNativeCredential) { + auto initial_conf = make_gcp_native_conf(); + auto rotated_conf = make_gcp_native_conf("rotated@my-project.iam.gserviceaccount.com"); + std::vector created_confs; + auto client = std::make_shared(std::shared_ptr {}); + S3ClientFactory::instance().set_client_creator_for_test( + [&](const S3ClientConf& conf) -> std::shared_ptr { + created_confs.push_back(conf); + return client; + }); + + io::ObjClientHolder holder(initial_conf); + ASSERT_TRUE(holder.init().ok()); + ASSERT_TRUE(holder.reset(rotated_conf).ok()); + + ASSERT_EQ(created_confs.size(), 2); + EXPECT_EQ(created_confs.back(), rotated_conf); + EXPECT_EQ(holder.s3_client_conf(), rotated_conf); +} + +TEST_F(S3ClientFactoryTest, ObjClientHolderResetSwitchesNativeAndStaticCredentials) { + auto native_conf = make_gcp_native_conf(); + auto static_conf = native_conf; + static_conf.credential = std::monostate {}; + static_conf.ak = "access-key"; + static_conf.sk = "secret-key"; + + std::vector created_confs; + auto client = std::make_shared(std::shared_ptr {}); + S3ClientFactory::instance().set_client_creator_for_test( + [&](const S3ClientConf& conf) -> std::shared_ptr { + created_confs.push_back(conf); + return client; + }); + + io::ObjClientHolder holder(native_conf); + ASSERT_TRUE(holder.init().ok()); + ASSERT_TRUE(holder.reset(static_conf).ok()); + ASSERT_TRUE(std::holds_alternative(holder.s3_client_conf().credential)); + EXPECT_EQ(holder.s3_client_conf().ak, "access-key"); + + ASSERT_TRUE(holder.reset(native_conf).ok()); + ASSERT_EQ(created_confs.size(), 3); + EXPECT_EQ(created_confs[1], static_conf); + EXPECT_EQ(created_confs[2], native_conf); + EXPECT_EQ(holder.s3_client_conf(), native_conf); +} + +TEST_F(S3ClientFactoryTest, CredentialTypeNamesAreExplicit) { + CredentialConfig none; + CredentialConfig gcp = GcpCredentialConfig {}; + + EXPECT_EQ(credential_type_name(none), "none"); + EXPECT_EQ(credential_type_name(gcp), "gcp"); +} + +TEST_F(S3ClientFactoryTest, GcpBearerHeaderSurvivesAnonymousAwsSigning) { + FixedTokenGcpS3Client client; + Aws::S3::Model::GetObjectRequest request; + request.SetBucket("bucket"); + request.SetKey("key"); + auto http_request = Aws::Http::CreateHttpRequest( + Aws::Http::URI("https://storage.googleapis.com/bucket/key"), + Aws::Http::HttpMethod::HTTP_GET, + Aws::Utils::Stream::DefaultResponseStreamFactoryMethod); + + client.BuildHttpRequest(request, http_request); + EXPECT_EQ(http_request->GetHeaderValue("Authorization"), "Bearer test-gcp-token"); + + // AWSAuthV4Signer reads this before it checks for anonymous credentials. + http_request->SetServiceSpecificParameters( + std::make_shared()); + Aws::Client::AWSAuthV4Signer signer( + std::make_shared(), "s3", "us-east-1"); + EXPECT_TRUE(signer.SignRequest(*http_request)); + EXPECT_EQ(http_request->GetHeaderValue("Authorization"), "Bearer test-gcp-token"); +} + +TEST_F(S3ClientFactoryTest, UnauthorizedObjectStoreResponseIsPermissionDenied) { + Aws::S3::S3Error error; + error.SetResponseCode(Aws::Http::HttpResponseCode::UNAUTHORIZED); + error.SetExceptionName("InvalidAuthenticationCredentials"); + error.SetMessage("missing or invalid bearer token"); + + auto status = doris::s3fs_error(error, "GCS request failed"); + EXPECT_EQ(status.code, ObjStorageStatus::PERMISSION_DENIED); +} + +TEST_F(S3ClientFactoryTest, GcpServiceAccountEmailValidation) { + const std::vector valid_emails = { + "target@my-project.iam.gserviceaccount.com", + "123456789-compute@developer.gserviceaccount.com", + "my-project@appspot.gserviceaccount.com", + }; + for (auto email : valid_emails) { + EXPECT_TRUE(is_valid_gcp_service_account_email(email)) << email; + } + + const std::vector invalid_emails = { + "", + "@my-project.iam.gserviceaccount.com", + "target@.iam.gserviceaccount.com", + "target@@my-project.iam.gserviceaccount.com", + "target@evil.developer.gserviceaccount.com", + "target@evil.appspot.gserviceaccount.com", + "target@my-project.iam.gserviceaccount.com.evil.example", + "target account@my-project.iam.gserviceaccount.com", + }; + for (auto email : invalid_emails) { + EXPECT_FALSE(is_valid_gcp_service_account_email(email)) << email; + } +} + TEST_F(S3ClientFactoryTest, AwsCredentialsProvider) { S3ClientFactory& factory = S3ClientFactory::instance(); S3ClientConf anonymous_conf; @@ -491,6 +637,156 @@ TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfProviderTypeMatrix) { } } +TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfGcpCredentialModes) { + S3URI s3_uri("s3://test-bucket/test-prefix"); + ASSERT_TRUE(s3_uri.parse().ok()); + + std::map base_properties { + {"AWS_ENDPOINT", "storage.googleapis.com"}, + {"AWS_REGION", "us-east1"}, + {"provider", "GCP"}, + }; + + struct TestCase { + const char* provider_type; + const char* impersonation_service_account; + GcpCredentialProviderType expected_provider_type; + }; + + // Both ADC and Compute Engine metadata credentials may be used directly or + // as the source credential for service-account impersonation. + std::vector cases = { + {"DEFAULT", "", GcpCredentialProviderType::Default}, + {"default", "", GcpCredentialProviderType::Default}, + {"DEFAULT", "target@my-project.iam.gserviceaccount.com", + GcpCredentialProviderType::Default}, + {"COMPUTE_ENGINE", "", GcpCredentialProviderType::ComputeEngine}, + {" compute_engine ", "", GcpCredentialProviderType::ComputeEngine}, + {"COMPUTE_ENGINE", "target@my-project.iam.gserviceaccount.com", + GcpCredentialProviderType::ComputeEngine}, + }; + + for (const auto& test_case : cases) { + auto properties = base_properties; + properties["gs.credential_provider_type"] = test_case.provider_type; + if (test_case.impersonation_service_account[0] != '\0') { + properties["gs.impersonation_service_account"] = + test_case.impersonation_service_account; + } + + S3Conf s3_conf; + ASSERT_TRUE( + S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf).ok()) + << "provider_type=" << test_case.provider_type; + ASSERT_EQ(s3_conf.client_conf.provider, io::ObjStorageProvider::GCP); + ASSERT_TRUE(std::holds_alternative(s3_conf.client_conf.credential)); + const auto& credential = std::get(s3_conf.client_conf.credential); + EXPECT_EQ(credential.provider_type, test_case.expected_provider_type); + EXPECT_EQ(credential.impersonation_service_account, + test_case.impersonation_service_account); + } +} + +TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfGcpDefaultImpersonation) { + S3URI s3_uri("s3://test-bucket/test-prefix"); + ASSERT_TRUE(s3_uri.parse().ok()); + std::map properties { + {"AWS_ENDPOINT", "storage.googleapis.com"}, + {"AWS_REGION", "us-east1"}, + {"provider", "GCP"}, + {"gs.impersonation_service_account", "target@my-project.iam.gserviceaccount.com"}, + }; + S3Conf s3_conf; + ASSERT_TRUE(S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf).ok()); + const auto& credential = std::get(s3_conf.client_conf.credential); + EXPECT_EQ(credential.provider_type, GcpCredentialProviderType::Default); + EXPECT_EQ(credential.impersonation_service_account, + "target@my-project.iam.gserviceaccount.com"); + + properties["gs.impersonation_service_account"] = ""; + ASSERT_TRUE(S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf).ok()); + const auto& direct_credential = std::get(s3_conf.client_conf.credential); + EXPECT_EQ(direct_credential.provider_type, GcpCredentialProviderType::Default); + EXPECT_TRUE(direct_credential.impersonation_service_account.empty()); +} + +TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfRejectsConflictingGcpCredentials) { + S3URI s3_uri("s3://test-bucket/test-prefix"); + ASSERT_TRUE(s3_uri.parse().ok()); + + std::map base_properties { + {"AWS_ENDPOINT", "storage.googleapis.com"}, + {"AWS_REGION", "us-east1"}, + {"provider", "GCP"}, + {"gs.credential_provider_type", "DEFAULT"}, + }; + + const std::vector> conflicts = { + {"AWS_ACCESS_KEY", "ak"}, + {"AWS_SECRET_KEY", "sk"}, + {"AWS_TOKEN", "token"}, + {"AWS_ROLE_ARN", "arn:aws:iam::123456789012:role/test-role"}, + {"AWS_EXTERNAL_ID", "external-id"}, + {"AWS_CREDENTIALS_PROVIDER_TYPE", "INSTANCE_PROFILE"}, + }; + for (const auto& [key, value] : conflicts) { + auto properties = base_properties; + properties[key] = value; + S3Conf s3_conf; + auto status = S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf); + ASSERT_FALSE(status.ok()) << key; + EXPECT_NE(status.to_string().find("cannot be combined"), std::string::npos) << key; + } +} + +TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfRejectsInvalidGcpCredentialModes) { + S3URI s3_uri("s3://test-bucket/test-prefix"); + ASSERT_TRUE(s3_uri.parse().ok()); + + std::map base_properties { + {"AWS_ENDPOINT", "storage.googleapis.com"}, + {"AWS_REGION", "us-east1"}, + {"provider", "GCP"}, + }; + for (const auto& provider_type : {"ADC", "", "unknown"}) { + auto properties = base_properties; + properties["gs.credential_provider_type"] = provider_type; + S3Conf s3_conf; + auto status = S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf); + ASSERT_FALSE(status.ok()) << provider_type; + EXPECT_NE(status.to_string().find("gs.credential_provider_type"), std::string::npos) + << provider_type; + EXPECT_NE(status.to_string().find("DEFAULT"), std::string::npos) << provider_type; + EXPECT_NE(status.to_string().find("COMPUTE_ENGINE"), std::string::npos) << provider_type; + } + + for (const auto& account : {" ", "invalid-account"}) { + auto properties = base_properties; + properties["gs.impersonation_service_account"] = account; + S3Conf s3_conf; + ASSERT_FALSE( + S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf).ok()); + } +} + +TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfRejectsGcpCredentialWithAwsProvider) { + S3URI s3_uri("s3://test-bucket/test-prefix"); + ASSERT_TRUE(s3_uri.parse().ok()); + + for (const auto& provider : {"S3", "AWS"}) { + std::map properties { + {"AWS_ENDPOINT", "storage.googleapis.com"}, + {"AWS_REGION", "us-east1"}, + {"provider", provider}, + {"gs.credential_provider_type", "DEFAULT"}, + }; + S3Conf s3_conf; + auto status = S3ClientFactory::convert_properties_to_s3_conf(properties, s3_uri, &s3_conf); + ASSERT_FALSE(status.ok()) << provider; + EXPECT_NE(status.to_string().find("provider=GCP"), std::string::npos) << provider; + } +} + TEST_F(S3ClientFactoryTest, ConvertPropertiesToS3ConfCredentialValidation) { S3URI s3_uri("s3://test-bucket/test-prefix"); ASSERT_TRUE(s3_uri.parse().ok()); diff --git a/cloud/CMakeLists.txt b/cloud/CMakeLists.txt index e0b44d9cdff1d2..32983430c60e05 100644 --- a/cloud/CMakeLists.txt +++ b/cloud/CMakeLists.txt @@ -139,6 +139,9 @@ set(GPERFTOOLS_HOME "${THIRDPARTY_DIR}/gperftools") # Set all libraries include(thirdparty) +set(absl_DIR ${THIRDPARTY_DIR}/lib/cmake/absl) +find_package(absl CONFIG REQUIRED PATHS ${absl_DIR} NO_DEFAULT_PATH) + find_program(THRIFT_COMPILER thrift ${CMAKE_SOURCE_DIR}/bin) # Check if functions are supported in this platform. All flags will generated @@ -321,6 +324,19 @@ else() add_definitions(-DUSE_LIBHDFS3) endif() +# google-cloud-cpp is linked as static archives in thirdparty.cmake, so its +# public Abseil dependencies must be present in the final link group as well. +set(COMMON_THIRDPARTY + ${COMMON_THIRDPARTY} + absl::base + absl::memory + absl::optional + absl::span + absl::str_format + absl::time + absl::variant +) + set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES} ${WL_START_GROUP} diff --git a/cloud/cmake/thirdparty.cmake b/cloud/cmake/thirdparty.cmake index a383b263b9c63d..d945ceaa45eb2f 100644 --- a/cloud/cmake/thirdparty.cmake +++ b/cloud/cmake/thirdparty.cmake @@ -72,6 +72,10 @@ add_thirdparty(brpc LIB64) add_thirdparty(rocksdb) # For local storage mocking add_thirdparty(libz LIBNAME "lib/libz.a") # Required by google::protobuf add_thirdparty(curl) +add_thirdparty(google_cloud_cpp_oauth2) +add_thirdparty(google_cloud_cpp_rest_internal) +add_thirdparty(google_cloud_cpp_common) +add_thirdparty(crc32c) add_thirdparty(zstd LIB64) add_thirdparty(fmt) # begin aws libs diff --git a/cloud/src/common/CMakeLists.txt b/cloud/src/common/CMakeLists.txt index a1b989cace1add..18d455561d2ae0 100644 --- a/cloud/src/common/CMakeLists.txt +++ b/cloud/src/common/CMakeLists.txt @@ -31,6 +31,7 @@ set(COMMON_FILES metric.cpp kms.cpp network_util.cpp + auth/obj_credential.cpp ../server/oss/cloud_server_starter_factory.cpp ) diff --git a/cloud/src/common/auth/obj_credential.cpp b/cloud/src/common/auth/obj_credential.cpp new file mode 100644 index 00000000000000..acb7116c1bc25c --- /dev/null +++ b/cloud/src/common/auth/obj_credential.cpp @@ -0,0 +1,239 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "common/auth/obj_credential.h" + +#include + +#include + +#include "cpp/obj-client/auth/obj_credential.h" + +namespace doris { +namespace { + +GcpCredentialConfig to_gcp_credential_config(const cloud::GcpCredentialPB& credential) { + GcpCredentialConfig config; + switch (credential.credential_provider_type()) { + case cloud::GcpCredentialPB::DEFAULT: + config.provider_type = GcpCredentialProviderType::Default; + break; + case cloud::GcpCredentialPB::COMPUTE_ENGINE: + config.provider_type = GcpCredentialProviderType::ComputeEngine; + break; + } + config.impersonation_service_account = credential.impersonation_service_account(); + return config; +} + +} // namespace + +void convert_obj_credential(const cloud::ObjectStoreInfoPB& source, CredentialConfig* credential) { + if (!source.has_credential() || !source.credential().has_gcp_credential()) { + *credential = std::monostate {}; + return; + } + *credential = to_gcp_credential_config(source.credential().gcp_credential()); +} + +} // namespace doris + +namespace doris::cloud { +namespace { + +// Extension checklist for a new passwordless provider: +// 1. add a provider-specific validate__obj_credential(); +// 2. dispatch it from validate_obj_credential(); and +// 3. reject requests that set more than one provider credential. +// Generic create/alter normalization does not need provider-specific branches. +ObjCredentialProvider to_obj_credential_provider(const ObjectStoreInfoPB& obj) { + if (!obj.has_provider()) { + return ObjCredentialProvider::Unknown; + } + switch (obj.provider()) { + case ObjectStoreInfoPB::S3: + return ObjCredentialProvider::Aws; + case ObjectStoreInfoPB::AZURE: + return ObjCredentialProvider::Azure; + case ObjectStoreInfoPB::BOS: + return ObjCredentialProvider::Bos; + case ObjectStoreInfoPB::COS: + return ObjCredentialProvider::Cos; + case ObjectStoreInfoPB::GCP: + return ObjCredentialProvider::Gcp; + case ObjectStoreInfoPB::OBS: + return ObjCredentialProvider::Obs; + case ObjectStoreInfoPB::OSS: + return ObjCredentialProvider::Oss; + case ObjectStoreInfoPB::TOS: + return ObjCredentialProvider::Tos; + case ObjectStoreInfoPB::UNKONWN: + return ObjCredentialProvider::Unknown; + } + return ObjCredentialProvider::Unknown; +} + +ObjCredentialValidationContext credential_validation_context(const ObjectStoreInfoPB& obj) { + return ObjCredentialValidationContext { + .provider = to_obj_credential_provider(obj), + .ak = obj.ak(), + .sk = obj.sk(), + .has_aws_role_arn = !obj.role_arn().empty(), + .has_aws_external_id = !obj.external_id().empty(), + .has_aws_credential_provider = obj.has_cred_provider_type(), + .has_encrypted_access_keys = obj.has_encryption_info(), + }; +} + +// A provider-native credential supersedes both common static credentials and +// provider-specific authentication fields outside the credential envelope. +void clear_legacy_credentials(ObjectStoreInfoPB* obj) { + obj->clear_ak(); + obj->clear_sk(); + obj->clear_encryption_info(); + obj->clear_cred_provider_type(); + obj->clear_role_arn(); + obj->clear_external_id(); +} + +std::optional validate_required_storage_fields(const ObjectStoreInfoPB& obj, + std::string_view provider) { + if (obj.bucket().empty() || obj.endpoint().empty() || obj.region().empty()) { + return std::string(provider) + " storage conf requires bucket, endpoint and region"; + } + return std::nullopt; +} + +std::optional validate_gcp_obj_credential(const ObjectStoreInfoPB& obj) { + const auto& credential = obj.credential().gcp_credential(); + if (!credential.has_credential_provider_type()) { + return "GCP credential provider type is required"; + } + if (auto error = validate_required_storage_fields(obj, "GCP"); error.has_value()) { + return error; + } + + CredentialConfig config; + convert_obj_credential(obj, &config); + return validate_obj_credential_config(config, credential_validation_context(obj)); +} + +} // namespace + +bool has_obj_credential(const ObjectStoreInfoPB& obj) { + return obj.has_credential(); +} + +std::optional validate_obj_credential(const ObjectStoreInfoPB& obj) { + if (!obj.has_credential()) { + return "credential is not set"; + } + + // Add one provider-specific branch for every new field in + // ObjectStoreCredentialPB. Reject multiple fields before dispatch when a + // second provider is introduced. The common visitor validates only the + // normalized credential itself. + if (obj.credential().has_gcp_credential()) { + return validate_gcp_obj_credential(obj); + } + return "unsupported object storage credential"; +} + +std::optional validate_obj_authentication(const ObjectStoreInfoPB& obj) { + if (has_obj_credential(obj)) { + return validate_obj_credential(obj); + } + if (obj.has_role_arn() && obj.role_arn().empty()) { + return "AWS role ARN cannot be empty"; + } + + const auto context = credential_validation_context(obj); + if (context.has_static_credentials() && context.has_aws_authentication()) { + return "access keys cannot be combined with AWS role or credential provider"; + } + if (!context.has_aws_authentication()) { + return std::nullopt; + } + if (!obj.has_provider() || obj.provider() != ObjectStoreInfoPB::S3) { + return "AWS role and credential provider require provider=S3"; + } + if (obj.has_external_id() && obj.external_id().size() > 0 && obj.role_arn().empty()) { + return "AWS external ID requires a role ARN"; + } + return std::nullopt; +} + +std::optional validate_and_normalize_obj_credential(ObjectStoreInfoPB* obj) { + if (auto error = validate_obj_credential(*obj); error.has_value()) { + return error; + } + clear_legacy_credentials(obj); + return std::nullopt; +} + +std::optional apply_obj_credential(const ObjectStoreInfoPB& update, + ObjectStoreInfoPB* target) { + if (!has_obj_credential(update)) { + return "credential is not set"; + } + if (credential_validation_context(update).has_conflicting_credentials()) { + return "credentials cannot be combined with other credentials"; + } + + // Merge into a copy first so a rejected alter never partially mutates target. + ObjectStoreInfoPB candidate = *target; + copy_obj_credential(update, &candidate); + if (update.credential().has_gcp_credential()) { + const auto& patch = update.credential().gcp_credential(); + if (!patch.has_credential_provider_type() && !patch.has_impersonation_service_account()) { + return "GCP credential update must specify a provider type or service account"; + } + auto* credential = candidate.mutable_credential()->mutable_gcp_credential(); + if (target->credential().has_gcp_credential()) { + credential->CopyFrom(target->credential().gcp_credential()); + credential->MergeFrom(patch); + } + // Only a new native credential needs a default; an existing source is + // preserved when the patch changes just the impersonation target. + if (!credential->has_credential_provider_type()) { + credential->set_credential_provider_type(GcpCredentialPB::DEFAULT); + } + if (credential->impersonation_service_account().empty()) { + credential->clear_impersonation_service_account(); + } + } + clear_legacy_credentials(&candidate); + if (auto error = validate_obj_credential(candidate); error.has_value()) { + return error; + } + + clear_legacy_credentials(target); + copy_obj_credential(candidate, target); + return std::nullopt; +} + +void copy_obj_credential(const ObjectStoreInfoPB& source, ObjectStoreInfoPB* target) { + if (&source == target) { + return; + } + target->clear_credential(); + if (source.has_credential()) { + target->mutable_credential()->CopyFrom(source.credential()); + } +} + +} // namespace doris::cloud diff --git a/cloud/src/common/auth/obj_credential.h b/cloud/src/common/auth/obj_credential.h new file mode 100644 index 00000000000000..a1ad7714e141a6 --- /dev/null +++ b/cloud/src/common/auth/obj_credential.h @@ -0,0 +1,65 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include + +namespace doris::cloud { + +// This helper keeps provider-specific credential branches out of MetaService. +// Provider authentication implementations belong in common/cpp/obj-client/auth; this +// Cloud layer owns vault-field validation, compatibility cleanup, and protobuf +// persistence mutations. + +class ObjectStoreInfoPB; + +// Whether a provider-native credential envelope is present. +bool has_obj_credential(const ObjectStoreInfoPB& obj); + +// TODO: Move AWS authentication fields into ObjectStoreInfoPB.credential so AWS +// follows the same validation and mutation path. Keep cred_provider_type, +// role_arn, and external_id in their current fields for now to preserve +// compatibility with stored vault metadata and existing clients. AK/SK remain +// the common static-credential representation for S3-compatible providers. + +// Validates the active ObjectStoreInfoPB.credential case without modifying obj. +// Add a provider-specific validation case when extending the credential envelope. +std::optional validate_obj_credential(const ObjectStoreInfoPB& obj); + +// Validates the complete authentication state after create/alter mutations. In +// addition to provider-native credentials, this rejects cross-provider AWS +// authentication and mutually exclusive authentication modes. +std::optional validate_obj_authentication(const ObjectStoreInfoPB& obj); + +// Used by create paths: validates first, then removes common static credentials +// and provider-specific authentication fields that must not coexist with the +// selected provider-native credential. +std::optional validate_and_normalize_obj_credential(ObjectStoreInfoPB* obj); + +// Used by alter paths: merges explicitly present GCP fields into the stored +// credential, with an empty service account clearing impersonation. Validates +// the result against target's storage fields before changing authentication. +std::optional apply_obj_credential(const ObjectStoreInfoPB& update, + ObjectStoreInfoPB* target); + +// Replaces only the credential envelope. An unset source clears target's credential; +// all non-credential fields are preserved. +void copy_obj_credential(const ObjectStoreInfoPB& source, ObjectStoreInfoPB* target); + +} // namespace doris::cloud diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index 0284d4246efcc1..fbdaae334fd185 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -31,6 +31,7 @@ #include #include +#include "common/auth/obj_credential.h" #include "common/bvars.h" #include "common/config.h" #include "common/encryption_util.h" @@ -765,7 +766,7 @@ static bool has_non_empty_role_arn(const ObjectStoreInfoPB& obj) { } static bool use_credential_provider(const ObjectStoreInfoPB& obj) { - return obj.has_cred_provider_type() || has_non_empty_role_arn(obj); + return obj.has_cred_provider_type() || has_non_empty_role_arn(obj) || has_obj_credential(obj); } static void create_object_info_with_encrypt(const InstanceInfoPB& instance, ObjectStoreInfoPB* obj, @@ -781,14 +782,23 @@ static void create_object_info_with_encrypt(const InstanceInfoPB& instance, Obje std::string external_endpoint = obj->has_external_endpoint() ? obj->external_endpoint() : ""; std::string region = obj->has_region() ? obj->region() : ""; - if (obj->has_role_arn()) { - if (obj->role_arn().empty() || !obj->has_cred_provider_type() || !obj->has_provider() || + if (has_obj_credential(*obj)) { + if (auto error = validate_and_normalize_obj_credential(obj); error.has_value()) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = *error; + return; + } + } else if (obj->has_role_arn()) { + if (obj->role_arn().empty() || !obj->has_provider() || obj->provider() != ObjectStoreInfoPB::S3 || bucket.empty() || endpoint.empty() || region.empty()) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "s3 conf info err with role_arn or cred provider, please check it"; return; } + if (!obj->has_cred_provider_type()) { + obj->set_cred_provider_type(CredProviderTypePB::INSTANCE_PROFILE); + } } else { // ATTN: prefix may be empty if (plain_ak.empty() || plain_sk.empty() || bucket.empty() || endpoint.empty() || @@ -1071,7 +1081,11 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt msg = ss.str(); return -1; } - auto* storage_vault_names = instance.mutable_storage_vault_names(); + // Apply all mutations to a candidate. The caller can safely abandon the + // transaction on any validation/serialization error without leaking a + // partial rename into the InstanceInfoPB it later persists. + InstanceInfoPB candidate_instance = instance; + auto* storage_vault_names = candidate_instance.mutable_storage_vault_names(); auto* name_ptr = storage_vault_names->Mutable(pos); DCHECK(name_ptr != nullptr); const std::string old_name = *name_ptr; @@ -1081,8 +1095,8 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt target_vault_id, name, old_name); return -1; } - std::string vault_id = instance.resource_ids().Get(pos); - auto vault_key = storage_vault_key({instance.instance_id(), vault_id}); + std::string vault_id = candidate_instance.resource_ids().Get(pos); + auto vault_key = storage_vault_key({candidate_instance.instance_id(), vault_id}); std::string val; auto err = txn->get(vault_key, &val); @@ -1118,7 +1132,7 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt return -1; } - if (vault_exist(instance, vault.alter_name())) { + if (vault_exist(candidate_instance, vault.alter_name())) { code = MetaServiceCode::ALREADY_EXISTED; msg = fmt::format("vault_name={} already existed", vault.alter_name()); return -1; @@ -1126,14 +1140,16 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt new_vault.set_name(vault.alter_name()); *name_ptr = vault.alter_name(); - if (instance.default_storage_vault_id() == vault_id) { - instance.set_default_storage_vault_name(vault.alter_name()); + if (candidate_instance.default_storage_vault_id() == vault_id) { + candidate_instance.set_default_storage_vault_name(vault.alter_name()); } } - if (obj_info.has_role_arn() && (obj_info.has_ak() || obj_info.has_sk())) { + if ((obj_info.has_role_arn() || obj_info.has_cred_provider_type() || + has_obj_credential(obj_info)) && + (obj_info.has_ak() || obj_info.has_sk())) { code = MetaServiceCode::INVALID_ARGUMENT; - msg = "invaild argument, both set ak/sk and role_arn is not allowed"; + msg = "invalid argument, access keys cannot be combined with role or credentials"; LOG(WARNING) << msg; return -1; } @@ -1162,28 +1178,68 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt new_vault.mutable_obj_info()->clear_role_arn(); new_vault.mutable_obj_info()->clear_external_id(); new_vault.mutable_obj_info()->clear_cred_provider_type(); + new_vault.mutable_obj_info()->clear_credential(); new_vault.mutable_obj_info()->set_ak(new_ak_sk_pair.first); new_vault.mutable_obj_info()->set_sk(new_ak_sk_pair.second); new_vault.mutable_obj_info()->mutable_encryption_info()->CopyFrom(encryption_info); } - if (obj_info.has_role_arn()) { + if (obj_info.has_role_arn() || obj_info.has_cred_provider_type()) { new_vault.mutable_obj_info()->clear_ak(); new_vault.mutable_obj_info()->clear_sk(); new_vault.mutable_obj_info()->clear_encryption_info(); + new_vault.mutable_obj_info()->clear_credential(); + + if (obj_info.has_role_arn()) { + if (obj_info.role_arn().empty()) { + new_vault.mutable_obj_info()->clear_role_arn(); + // An external ID has no meaning without a role. Clearing the role + // explicitly clears its dependent value as well. + new_vault.mutable_obj_info()->clear_external_id(); + } else { + new_vault.mutable_obj_info()->set_role_arn(obj_info.role_arn()); + if (!new_vault.obj_info().has_cred_provider_type() && + !obj_info.has_cred_provider_type()) { + new_vault.mutable_obj_info()->set_cred_provider_type( + CredProviderTypePB::INSTANCE_PROFILE); + } + } + } + if (obj_info.has_cred_provider_type()) { + new_vault.mutable_obj_info()->set_cred_provider_type(obj_info.cred_provider_type()); + } + } - new_vault.mutable_obj_info()->set_role_arn(obj_info.role_arn()); - new_vault.mutable_obj_info()->set_cred_provider_type(get_cred_provider_type(obj_info)); - if (obj_info.has_external_id()) { + // external_id is an independent ALTER field. It may be changed without + // requiring FE to resend role_arn and cred_provider_type. + if (obj_info.has_external_id()) { + if (obj_info.external_id().empty()) { + new_vault.mutable_obj_info()->clear_external_id(); + } else { new_vault.mutable_obj_info()->set_external_id(obj_info.external_id()); } } + if (has_obj_credential(obj_info)) { + if (auto error = apply_obj_credential(obj_info, new_vault.mutable_obj_info()); + error.has_value()) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = *error; + return -1; + } + } + if (obj_info.has_use_path_style()) { new_vault.mutable_obj_info()->set_use_path_style(obj_info.use_path_style()); } + if (auto error = validate_obj_authentication(new_vault.obj_info()); error.has_value()) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = *error; + return -1; + } + auto now_time = std::chrono::system_clock::now(); uint64_t time = std::chrono::duration_cast(now_time.time_since_epoch()).count(); @@ -1198,6 +1254,7 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt } txn->put(vault_key, val); + instance.Swap(&candidate_instance); LOG(INFO) << "put vault_id=" << vault_id << ", vault_key=" << hex(vault_key) << ", origin vault=" << encryt_sk(hide_ak(origin_vault_info)) << ", new vault=" << encryt_sk(hide_ak(new_vault_info)); @@ -1259,7 +1316,7 @@ static int extract_object_storage_info(const AlterObjStoreInfoRequest* request, external_id] = obj_desc; bool use_credential_provider_for_add_vault = - request->op() == AlterObjStoreInfoRequest::ADD_S3_VAULT && obj.has_cred_provider_type(); + request->op() == AlterObjStoreInfoRequest::ADD_S3_VAULT && use_credential_provider(obj); if (!obj.has_role_arn() && !use_credential_provider_for_add_vault) { if (!obj.has_ak() || !obj.has_sk()) { code = MetaServiceCode::INVALID_ARGUMENT; @@ -1325,7 +1382,10 @@ static ObjectStoreInfoPB object_info_pb_factory(ObjectStorageDesc& obj_desc, last_item.set_role_arn(role_arn); last_item.set_external_id(external_id); } - last_item.set_cred_provider_type(get_cred_provider_type(obj)); + if (obj.has_cred_provider_type() || has_non_empty_role_arn(obj)) { + last_item.set_cred_provider_type(get_cred_provider_type(obj)); + } + copy_obj_credential(obj, &last_item); } last_item.set_bucket(bucket); // format prefix, such as `/aa/bb/`, `aa/bb//`, `//aa/bb`, ` /aa/bb` -> `aa/bb` @@ -1496,7 +1556,11 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr } if (use_credential_provider(obj)) { - if (!obj.has_provider() || obj.provider() != ObjectStoreInfoPB::S3) { + bool valid_aws = obj.has_cred_provider_type() && !has_obj_credential(obj) && + obj.has_provider() && obj.provider() == ObjectStoreInfoPB::S3; + bool valid_credential = + has_obj_credential(obj) && !validate_obj_credential(obj).has_value(); + if (!valid_aws && !valid_credential) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "s3 conf info err with credentials_provider_type, please check it"; return; @@ -1620,7 +1684,9 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr break; } case AlterObjStoreInfoRequest::ALTER_S3_VAULT: { - alter_s3_storage_vault(instance, txn, request->vault(), code, msg, response); + if (alter_s3_storage_vault(instance, txn, request->vault(), code, msg, response) != 0) { + return; + } break; } case AlterObjStoreInfoRequest::ALTER_HDFS_VAULT: { diff --git a/cloud/src/recycler/s3_accessor.cpp b/cloud/src/recycler/s3_accessor.cpp index 50bf53703aa10e..6013526783cc3a 100644 --- a/cloud/src/recycler/s3_accessor.cpp +++ b/cloud/src/recycler/s3_accessor.cpp @@ -48,6 +48,8 @@ #include "cpp/obj-client/azure_obj_storage_client.h" #endif #include "cpp/aws_logger.h" +#include "cpp/obj-client/auth/obj_credential.h" +#include "cpp/obj-client/auth/obj_s3_client_factory.h" #include "cpp/obj-client/rate_limited_obj_storage_client.h" #include "cpp/obj-client/s3_obj_storage_client.h" #include "cpp/obj_retry_strategy.h" @@ -301,6 +303,8 @@ std::optional S3Conf::from_obj_store_info(const ObjectStoreInfoPB& obj_i } } + convert_obj_credential(obj_info, &s3_conf.credential); + s3_conf.endpoint = obj_info.endpoint(); s3_conf.region = obj_info.region(); s3_conf.bucket = obj_info.bucket(); @@ -468,10 +472,16 @@ int S3Accessor::init() { LOG(WARNING) << "failed to create AWS credential provider: " << credentials.error; return -1; } - auto s3_client = std::make_shared( - std::move(credentials.provider), std::move(aws_config), - Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, - conf_.use_virtual_addressing /* useVirtualAddressing */); + auto s3_client = doris::make_s3_client( + conf_.credential, + doris::S3ClientBuildContext { + .fallback_provider = std::move(credentials.provider), + .config = std::move(aws_config), + .payload_signing_policy = + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, + .use_virtual_addressing = conf_.use_virtual_addressing, + .ca_cert_path = _ca_cert_file_path, + }); auto client = std::make_shared(std::move(s3_client), ObjStorageEndpointInfo { .endpoint = conf_.endpoint, diff --git a/cloud/src/recycler/s3_accessor.h b/cloud/src/recycler/s3_accessor.h index deded63abe1796..adee55e2195b0c 100644 --- a/cloud/src/recycler/s3_accessor.h +++ b/cloud/src/recycler/s3_accessor.h @@ -22,9 +22,11 @@ #include #include #include +#include #include "cpp/aws_common.h" #include "cpp/obj-client/auth/aws_credential_factory.h" +#include "cpp/obj-client/auth/obj_credential.h" #include "cpp/obj-client/obj_storage_client.h" #include "recycler/storage_vault_accessor.h" @@ -87,6 +89,7 @@ struct S3Conf { CredProviderType cred_provider_type = CredProviderType::Default; std::string role_arn; std::string external_id; + doris::CredentialConfig credential {}; enum Provider : uint8_t { S3, diff --git a/cloud/test/CMakeLists.txt b/cloud/test/CMakeLists.txt index 6620b0184fe726..fa3c031b6040c8 100644 --- a/cloud/test/CMakeLists.txt +++ b/cloud/test/CMakeLists.txt @@ -80,6 +80,8 @@ add_executable(log_test log_test.cpp) add_executable(resource_test resource_test.cpp) +add_executable(obj_credential_test obj_credential_test.cpp) + add_executable(rpc_kv_bvar_test rpc_kv_bvar_test.cpp) add_executable(http_encode_key_test http_encode_key_test.cpp) @@ -175,6 +177,8 @@ target_link_libraries(log_test ${TEST_LINK_LIBS}) target_link_libraries(resource_test ${TEST_LINK_LIBS}) +target_link_libraries(obj_credential_test ${TEST_LINK_LIBS}) + target_link_libraries(rpc_kv_bvar_test ${TEST_LINK_LIBS}) target_link_libraries(http_encode_key_test ${TEST_LINK_LIBS}) diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index b32813071f06ab..d5ef2b246d16ba 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -35,6 +35,7 @@ #include #include #include +#include #include #include "common/config.h" @@ -734,10 +735,28 @@ TEST(MetaServiceTest, AlterS3StorageVaultTest) { InstanceInfoPB instance; instance.add_storage_vault_names(vault.name()); instance.add_resource_ids(vault.id()); + + constexpr char gcp_vault_name[] = "test_alter_gcp_vault"; + StorageVaultPB gcp_vault; + gcp_vault.set_name(gcp_vault_name); + gcp_vault.set_id("3"); + auto* gcp_obj_info = gcp_vault.mutable_obj_info(); + gcp_obj_info->set_id("3"); + gcp_obj_info->set_provider(ObjectStoreInfoPB::GCP); + gcp_obj_info->set_bucket("gcp-bucket"); + gcp_obj_info->set_endpoint("storage.googleapis.com"); + gcp_obj_info->set_region("us-east1"); + gcp_obj_info->mutable_credential()->mutable_gcp_credential()->set_credential_provider_type( + GcpCredentialPB::COMPUTE_ENGINE); + gcp_obj_info->mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + "original@my-project.iam.gserviceaccount.com"); + instance.add_storage_vault_names(gcp_vault.name()); + instance.add_resource_ids(gcp_vault.id()); instance.set_instance_id("GetObjStoreInfoTestInstance"); val = instance.SerializeAsString(); txn->put(key, val); txn->put(storage_vault_key({instance.instance_id(), "2"}), vault.SerializeAsString()); + txn->put(storage_vault_key({instance.instance_id(), "3"}), gcp_vault.SerializeAsString()); ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK); txn = nullptr; @@ -752,6 +771,173 @@ TEST(MetaServiceTest, AlterS3StorageVaultTest) { i.ParseFromString(val); }; + auto get_test_vault = [&](std::string_view vault_id, StorageVaultPB& stored_vault) { + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({instance.instance_id(), vault_id}), &val), + TxnErrorCode::TXN_OK); + ASSERT_TRUE(stored_vault.ParseFromString(val)); + }; + + // A rejected credential update must not commit a rename that was applied + // earlier in the request. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(vault_name); + req.mutable_vault()->set_alter_name("rename_must_not_commit"); + req.mutable_vault() + ->mutable_obj_info() + ->mutable_credential() + ->mutable_gcp_credential() + ->set_credential_provider_type(GcpCredentialPB::DEFAULT); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT) << res.status().msg(); + + InstanceInfoPB stored_instance; + get_test_instance(stored_instance); + ASSERT_EQ(stored_instance.storage_vault_names(0), vault_name); + StorageVaultPB stored_vault; + get_test_vault("2", stored_vault); + ASSERT_EQ(stored_vault.name(), vault_name); + } + + // A GCP-native vault cannot be changed to either AWS role or AWS + // credential-provider authentication while its provider remains GCP. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(gcp_vault_name); + auto* update = req.mutable_vault()->mutable_obj_info(); + update->set_role_arn("arn:aws:iam::123456789012:role/test-role"); + update->set_cred_provider_type(CredProviderTypePB::INSTANCE_PROFILE); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT) << res.status().msg(); + ASSERT_NE(res.status().msg().find("provider=S3"), std::string::npos); + } + + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(gcp_vault_name); + req.mutable_vault()->mutable_obj_info()->set_cred_provider_type( + CredProviderTypePB::CONTAINER); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT) << res.status().msg(); + ASSERT_NE(res.status().msg().find("provider=S3"), std::string::npos); + + InstanceInfoPB stored_instance; + get_test_instance(stored_instance); + ASSERT_EQ(stored_instance.storage_vault_names(1), gcp_vault_name); + StorageVaultPB stored_vault; + get_test_vault("3", stored_vault); + ASSERT_TRUE(stored_vault.obj_info().has_credential()); + ASSERT_TRUE(stored_vault.obj_info().credential().has_gcp_credential()); + ASSERT_FALSE(stored_vault.obj_info().has_role_arn()); + ASSERT_FALSE(stored_vault.obj_info().has_cred_provider_type()); + } + + // Verify partial GCP updates at the RPC and persistence boundary. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(gcp_vault_name); + auto alter_gcp = [&]() { + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, + nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + }; + auto* update = req.mutable_vault()->mutable_obj_info(); + auto* patch = update->mutable_credential()->mutable_gcp_credential(); + patch->set_impersonation_service_account("updated@my-project.iam.gserviceaccount.com"); + alter_gcp(); + StorageVaultPB stored; + get_test_vault("3", stored); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().credential_provider_type(), + GcpCredentialPB::COMPUTE_ENGINE); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().impersonation_service_account(), + "updated@my-project.iam.gserviceaccount.com"); + + patch->Clear(); + patch->set_credential_provider_type(GcpCredentialPB::DEFAULT); + alter_gcp(); + get_test_vault("3", stored); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().credential_provider_type(), + GcpCredentialPB::DEFAULT); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().impersonation_service_account(), + "updated@my-project.iam.gserviceaccount.com"); + + patch->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + alter_gcp(); + patch->Clear(); + patch->set_impersonation_service_account(""); + alter_gcp(); + get_test_vault("3", stored); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().credential_provider_type(), + GcpCredentialPB::COMPUTE_ENGINE); + ASSERT_FALSE(stored.obj_info() + .credential() + .gcp_credential() + .has_impersonation_service_account()); + + // Rejected patches must preserve both the stored vault and its name. + const auto original = stored.SerializeAsString(); + patch->set_impersonation_service_account("invalid-account"); + req.mutable_vault()->set_alter_name("invalid_gcp_rename"); + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT); + get_test_vault("3", stored); + ASSERT_EQ(stored.SerializeAsString(), original); + InstanceInfoPB stored_instance; + get_test_instance(stored_instance); + ASSERT_EQ(stored_instance.storage_vault_names(1), gcp_vault_name); + req.mutable_vault()->clear_alter_name(); + + // Switching from encrypted static keys creates a new native credential. + update->Clear(); + update->set_ak("gcp-ak"); + update->set_sk("gcp-sk"); + alter_gcp(); + get_test_vault("3", stored); + ASSERT_FALSE(stored.obj_info().has_credential()); + ASSERT_TRUE(stored.obj_info().has_encryption_info()); + update->Clear(); + update->mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + "updated@my-project.iam.gserviceaccount.com"); + alter_gcp(); + get_test_vault("3", stored); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().credential_provider_type(), + GcpCredentialPB::DEFAULT); + ASSERT_EQ(stored.obj_info().credential().gcp_credential().impersonation_service_account(), + "updated@my-project.iam.gserviceaccount.com"); + ASSERT_FALSE(stored.obj_info().has_ak()); + ASSERT_FALSE(stored.obj_info().has_sk()); + ASSERT_FALSE(stored.obj_info().has_encryption_info()); + } + { AlterObjStoreInfoRequest req; req.set_cloud_unique_id("test_cloud_unique_id"); @@ -12595,6 +12781,87 @@ TEST(MetaServiceTest, AlterS3StorageVaultWithRoleArnTest) { ASSERT_EQ(get_obj.name(), new_vault_name) << get_obj.obj_info().ShortDebugString(); } + // Role, external ID, and credential provider are independent ALTER fields. + // Updating one must preserve the other two. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name("new_test_alter_s3_vault_111"); + req.mutable_vault()->mutable_obj_info()->set_cred_provider_type( + CredProviderTypePB::CONTAINER); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({"GetObjStoreInfoTestInstance", "2"}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB stored_vault; + ASSERT_TRUE(stored_vault.ParseFromString(val)); + ASSERT_EQ(stored_vault.obj_info().role_arn(), "arn:aws:iam::12311321:role/test-alter-role"); + ASSERT_EQ(stored_vault.obj_info().external_id(), "external_id_123123"); + ASSERT_EQ(stored_vault.obj_info().cred_provider_type(), CredProviderTypePB::CONTAINER); + } + + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name("new_test_alter_s3_vault_111"); + req.mutable_vault()->mutable_obj_info()->set_role_arn( + "arn:aws:iam::12311321:role/replacement-role"); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({"GetObjStoreInfoTestInstance", "2"}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB stored_vault; + ASSERT_TRUE(stored_vault.ParseFromString(val)); + ASSERT_EQ(stored_vault.obj_info().role_arn(), + "arn:aws:iam::12311321:role/replacement-role"); + ASSERT_EQ(stored_vault.obj_info().external_id(), "external_id_123123"); + ASSERT_EQ(stored_vault.obj_info().cred_provider_type(), CredProviderTypePB::CONTAINER); + } + + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name("new_test_alter_s3_vault_111"); + req.mutable_vault()->mutable_obj_info()->set_external_id("replacement-external-id"); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({"GetObjStoreInfoTestInstance", "2"}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB stored_vault; + ASSERT_TRUE(stored_vault.ParseFromString(val)); + ASSERT_EQ(stored_vault.obj_info().role_arn(), + "arn:aws:iam::12311321:role/replacement-role"); + ASSERT_EQ(stored_vault.obj_info().external_id(), "replacement-external-id"); + ASSERT_EQ(stored_vault.obj_info().cred_provider_type(), CredProviderTypePB::CONTAINER); + } + std::string cipher_sk = "JUkuTDctR+ckJtnPkLScWaQZRcOtWBhsLLpnCRxQLxr734qB8cs6gNLH6grE1FxO"; std::string plain_sk = "Hx60p12123af234541nsVsffdfsdfghsdfhsdf34t"; diff --git a/cloud/test/obj_credential_test.cpp b/cloud/test/obj_credential_test.cpp new file mode 100644 index 00000000000000..50905efec6dd48 --- /dev/null +++ b/cloud/test/obj_credential_test.cpp @@ -0,0 +1,149 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "common/auth/obj_credential.h" + +#include +#include + +namespace doris::cloud { +namespace { + +constexpr char ACCOUNT[] = "original@my-project.iam.gserviceaccount.com"; +constexpr char NEW_ACCOUNT[] = "updated@my-project.iam.gserviceaccount.com"; + +ObjectStoreInfoPB make_gcp_vault() { + ObjectStoreInfoPB obj; + obj.set_provider(ObjectStoreInfoPB::GCP); + obj.set_bucket("test-bucket"); + obj.set_endpoint("storage.googleapis.com"); + obj.set_region("us-east1"); + auto* credential = obj.mutable_credential()->mutable_gcp_credential(); + credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + credential->set_impersonation_service_account(ACCOUNT); + return obj; +} + +TEST(ObjCredentialTest, AlterTargetPreservesSourceAndStorageFields) { + auto stored = make_gcp_vault(); + auto expected = stored; + expected.mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + NEW_ACCOUNT); + ObjectStoreInfoPB update; + update.mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + NEW_ACCOUNT); + auto error = apply_obj_credential(update, &stored); + ASSERT_FALSE(error.has_value()) << error.value_or(""); + EXPECT_EQ(stored.SerializeAsString(), expected.SerializeAsString()); +} + +TEST(ObjCredentialTest, AlterSourcePreservesTarget) { + for (auto source : {GcpCredentialPB::COMPUTE_ENGINE, GcpCredentialPB::DEFAULT}) { + auto stored = make_gcp_vault(); + ObjectStoreInfoPB update; + update.mutable_credential()->mutable_gcp_credential()->set_credential_provider_type(source); + auto error = apply_obj_credential(update, &stored); + ASSERT_FALSE(error.has_value()) << error.value_or(""); + EXPECT_EQ(stored.credential().gcp_credential().credential_provider_type(), source); + EXPECT_EQ(stored.credential().gcp_credential().impersonation_service_account(), ACCOUNT); + } +} + +TEST(ObjCredentialTest, ExplicitEmptyTargetClearsOnlyImpersonation) { + auto stored = make_gcp_vault(); + ObjectStoreInfoPB update; + update.mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account(""); + auto error = apply_obj_credential(update, &stored); + ASSERT_FALSE(error.has_value()) << error.value_or(""); + EXPECT_EQ(stored.credential().gcp_credential().credential_provider_type(), + GcpCredentialPB::COMPUTE_ENGINE); + EXPECT_FALSE(stored.credential().gcp_credential().has_impersonation_service_account()); +} + +TEST(ObjCredentialTest, SwitchFromStaticDefaultsSourceAndRemovesLegacyCredentials) { + for (auto account : {NEW_ACCOUNT, ""}) { + for (bool explicit_source : {false, true}) { + auto stored = make_gcp_vault(); + stored.clear_credential(); + stored.set_ak("encrypted-ak"); + stored.set_sk("encrypted-sk"); + stored.mutable_encryption_info()->set_key_id(1); + ObjectStoreInfoPB update; + auto* credential = update.mutable_credential()->mutable_gcp_credential(); + credential->set_impersonation_service_account(account); + if (explicit_source) { + credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + } + auto error = apply_obj_credential(update, &stored); + ASSERT_FALSE(error.has_value()) << error.value_or(""); + EXPECT_TRUE(stored.credential().gcp_credential().has_credential_provider_type()); + EXPECT_EQ(stored.credential().gcp_credential().credential_provider_type(), + explicit_source ? GcpCredentialPB::COMPUTE_ENGINE : GcpCredentialPB::DEFAULT); + EXPECT_EQ(stored.credential().gcp_credential().impersonation_service_account(), + account); + EXPECT_FALSE(stored.has_ak()); + EXPECT_FALSE(stored.has_sk()); + EXPECT_FALSE(stored.has_encryption_info()); + EXPECT_FALSE(stored.has_cred_provider_type()); + EXPECT_FALSE(stored.has_role_arn()); + EXPECT_FALSE(stored.has_external_id()); + } + } +} + +TEST(ObjCredentialTest, RejectedPatchDoesNotMutateStoredCredential) { + for (bool native : {false, true}) { + auto stored = make_gcp_vault(); + if (!native) { + stored.clear_credential(); + stored.set_ak("encrypted-ak"); + stored.set_sk("encrypted-sk"); + stored.mutable_encryption_info()->set_key_id(1); + } + const auto original = stored.SerializeAsString(); + ObjectStoreInfoPB update; + update.mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + "invalid-account"); + EXPECT_TRUE(apply_obj_credential(update, &stored).has_value()); + EXPECT_EQ(stored.SerializeAsString(), original); + + update.mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + NEW_ACCOUNT); + update.set_ak("conflicting-ak"); + EXPECT_TRUE(apply_obj_credential(update, &stored).has_value()); + EXPECT_EQ(stored.SerializeAsString(), original); + } +} + +TEST(ObjCredentialTest, RejectsEmptyPatchAndCrossProviderUpdate) { + auto stored = make_gcp_vault(); + auto original = stored.SerializeAsString(); + ObjectStoreInfoPB update; + update.mutable_credential()->mutable_gcp_credential(); + EXPECT_TRUE(apply_obj_credential(update, &stored).has_value()); + EXPECT_EQ(stored.SerializeAsString(), original); + + stored.set_provider(ObjectStoreInfoPB::S3); + original = stored.SerializeAsString(); + update.mutable_credential()->mutable_gcp_credential()->set_impersonation_service_account( + NEW_ACCOUNT); + EXPECT_TRUE(apply_obj_credential(update, &stored).has_value()); + EXPECT_EQ(stored.SerializeAsString(), original); +} + +} // namespace +} // namespace doris::cloud diff --git a/cloud/test/s3_accessor_test.cpp b/cloud/test/s3_accessor_test.cpp index fa2bfd1c7efe6e..ef69a0d45ab647 100644 --- a/cloud/test/s3_accessor_test.cpp +++ b/cloud/test/s3_accessor_test.cpp @@ -27,6 +27,7 @@ #include #include +#include "common/auth/obj_credential.h" #include "common/config.h" #include "common/configbase.h" #include "common/logging.h" @@ -396,6 +397,44 @@ TEST_F(S3AccessorTest, path_style_test) { } } +TEST(S3AccessorConfigTest, GcpCredentialFromObjectStoreInfo) { + ObjectStoreInfoPB obj_info; + obj_info.set_provider(ObjectStoreInfoPB::GCP); + obj_info.set_endpoint("storage.googleapis.com"); + obj_info.set_region("us-east1"); + obj_info.set_bucket("test-bucket"); + auto* gcp_credential = obj_info.mutable_credential()->mutable_gcp_credential(); + gcp_credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + gcp_credential->set_impersonation_service_account("target@my-project.iam.gserviceaccount.com"); + + auto s3_conf = S3Conf::from_obj_store_info(obj_info); + ASSERT_TRUE(s3_conf.has_value()); + EXPECT_EQ(s3_conf->provider, S3Conf::GCS); + ASSERT_TRUE(std::holds_alternative(s3_conf->credential)); + const auto& credential = std::get(s3_conf->credential); + EXPECT_EQ(credential.provider_type, doris::GcpCredentialProviderType::ComputeEngine); + EXPECT_EQ(credential.impersonation_service_account, + "target@my-project.iam.gserviceaccount.com"); +} + +TEST(S3AccessorConfigTest, ValidateGcpStorageVaultCredential) { + ObjectStoreInfoPB obj_info; + obj_info.set_provider(ObjectStoreInfoPB::GCP); + obj_info.set_endpoint("storage.googleapis.com"); + obj_info.set_region("us-east1"); + obj_info.set_bucket("test-bucket"); + obj_info.mutable_credential()->mutable_gcp_credential()->set_credential_provider_type( + GcpCredentialPB::DEFAULT); + + EXPECT_FALSE(validate_obj_credential(obj_info).has_value()); + + obj_info.set_ak("ak"); + obj_info.set_sk("sk"); + auto error = validate_obj_credential(obj_info); + ASSERT_TRUE(error.has_value()); + EXPECT_NE(error->find("cannot be combined"), std::string::npos); +} + class S3AccessorRoleTest : public testing::Test { static void SetUpTestSuite() { if (!std::getenv("AWS_ROLE_ARN") || !std::getenv("AWS_EXTERNAL_ID") || diff --git a/common/cpp/obj-client/auth/gcp/gcp_auth.cpp b/common/cpp/obj-client/auth/gcp/gcp_auth.cpp new file mode 100644 index 00000000000000..801b7b5c7f6836 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcp_auth.cpp @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/gcp/gcp_auth.h" + +#include +#include +#include + +namespace doris { + +bool is_valid_gcp_service_account_email(std::string_view email) { + constexpr std::string_view user_managed_suffix = ".iam.gserviceaccount.com"; + constexpr std::string_view compute_engine_domain = "developer.gserviceaccount.com"; + constexpr std::string_view app_engine_domain = "appspot.gserviceaccount.com"; + auto at = email.find('@'); + if (at == std::string_view::npos || at == 0 || + email.find('@', at + 1) != std::string_view::npos || + std::any_of(email.begin(), email.end(), [](unsigned char c) { return std::isspace(c); })) { + return false; + } + + auto domain = email.substr(at + 1); + return domain == compute_engine_domain || domain == app_engine_domain || + (domain.size() > user_managed_suffix.size() && domain.ends_with(user_managed_suffix)); +} + +std::optional parse_gcp_credential_provider_type( + std::string_view provider_type) { + std::string normalized(provider_type); + normalized.erase(normalized.begin(), + std::find_if(normalized.begin(), normalized.end(), + [](unsigned char c) { return !std::isspace(c); })); + normalized.erase(std::find_if(normalized.rbegin(), normalized.rend(), + [](unsigned char c) { return !std::isspace(c); }) + .base(), + normalized.end()); + std::transform(normalized.begin(), normalized.end(), normalized.begin(), + [](unsigned char c) { return static_cast(std::toupper(c)); }); + if (normalized == "DEFAULT") { + return GcpCredentialProviderType::Default; + } + if (normalized == "COMPUTE_ENGINE") { + return GcpCredentialProviderType::ComputeEngine; + } + return std::nullopt; +} + +std::optional validate_gcp_credential(const GcpCredentialConfig& credential) { + if (credential.impersonation_service_account.empty()) { + return std::nullopt; + } + if (!is_valid_gcp_service_account_email(credential.impersonation_service_account)) { + return "Invalid GCP service account email"; + } + return std::nullopt; +} + +GcpCredentialParseResult parse_gcp_credential_properties( + const ObjCredentialPropertyGetter& get_property) { + auto provider_type_value = get_property(GCP_CREDENTIAL_PROVIDER_TYPE); + auto impersonation_service_account = get_property(GCP_IMPERSONATION_SERVICE_ACCOUNT); + if (!provider_type_value.has_value() && !impersonation_service_account.has_value()) { + return {}; + } + auto provider_type = provider_type_value.has_value() + ? parse_gcp_credential_provider_type(*provider_type_value) + : GcpCredentialProviderType::Default; + if (!provider_type.has_value()) { + return {.error = "invalid " + std::string(GCP_CREDENTIAL_PROVIDER_TYPE) + " value \"" + + *provider_type_value + + "\"; supported values are DEFAULT and COMPUTE_ENGINE"}; + } + + GcpCredentialConfig config {.provider_type = *provider_type}; + if (impersonation_service_account.has_value()) { + config.impersonation_service_account = std::move(*impersonation_service_account); + } + if (auto error = validate_gcp_credential(config); error.has_value()) { + return {.error = std::move(error)}; + } + return {.credential = std::move(config)}; +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_auth.h b/common/cpp/obj-client/auth/gcp/gcp_auth.h new file mode 100644 index 00000000000000..99b35f7be5049b --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcp_auth.h @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include "cpp/obj-client/auth/obj_credential_common.h" + +namespace doris { + +// Provider-specific authentication lives under cpp/obj-client/auth/. Future +// providers should use sibling packages and expose only their normalized config, +// validation, and client-setup entry points there. + +inline constexpr std::string_view GCP_CREDENTIAL_PROVIDER_TYPE = "gs.credential_provider_type"; +inline constexpr std::string_view GCP_IMPERSONATION_SERVICE_ACCOUNT = + "gs.impersonation_service_account"; + +enum class GcpCredentialProviderType { + Default, + ComputeEngine, +}; + +struct GcpCredentialConfig { + GcpCredentialProviderType provider_type = GcpCredentialProviderType::Default; + std::string impersonation_service_account {}; + + bool operator==(const GcpCredentialConfig&) const = default; +}; + +struct GcpCredentialParseResult { + std::optional credential = std::nullopt; + std::optional error = std::nullopt; +}; + +bool is_valid_gcp_service_account_email(std::string_view email); +std::optional parse_gcp_credential_provider_type( + std::string_view provider_type); +std::optional validate_gcp_credential(const GcpCredentialConfig& credential); +GcpCredentialParseResult parse_gcp_credential_properties( + const ObjCredentialPropertyGetter& get_property); + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp new file mode 100644 index 00000000000000..0ad7ba19d2ce26 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp @@ -0,0 +1,51 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/gcp/gcp_s3_client.h" + +#include + +#include + +namespace doris { + +GcpS3Client::GcpS3Client(const GcpCredentialConfig& credential, std::string ca_cert_path, + Aws::Client::ClientConfiguration aws_config, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy payload_signing_policy, + bool use_virtual_addressing) + : Aws::S3::S3Client(std::make_shared(), + std::move(aws_config), payload_signing_policy, use_virtual_addressing), + _token_provider(credential, ca_cert_path) {} + +void GcpS3Client::BuildHttpRequest( + const Aws::AmazonWebServiceRequest& request, + const std::shared_ptr& http_request) const { + Aws::S3::S3Client::BuildHttpRequest(request, http_request); + auto token = fetch_token(); + if (!token.has_value()) { + // GCS will return a useful 401 response. The token provider has already + // logged the underlying credential or metadata error. + return; + } + http_request->SetHeaderValue("Authorization", Aws::String("Bearer ") + token->c_str()); +} + +std::optional GcpS3Client::fetch_token() const { + return _token_provider.get_token(); +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.h b/common/cpp/obj-client/auth/gcp/gcp_s3_client.h new file mode 100644 index 00000000000000..4e15f3255b3c78 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.h @@ -0,0 +1,55 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include +#include +#include + +#include "cpp/obj-client/auth/gcp/gcp_auth.h" +#include "cpp/obj-client/auth/gcp/gcp_token_provider.h" + +namespace doris { + +// GCS accepts OAuth2 bearer tokens on its S3-compatible XML API. The client uses +// anonymous AWS credentials so AWSAuthV4Signer returns without adding SigV4, and +// BuildHttpRequest injects the bearer header immediately before that signing step. +class GcpS3Client : public Aws::S3::S3Client { +public: + GcpS3Client(const GcpCredentialConfig& credential, std::string ca_cert_path, + Aws::Client::ClientConfiguration aws_config, + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy payload_signing_policy, + bool use_virtual_addressing); + +protected: + void BuildHttpRequest( + const Aws::AmazonWebServiceRequest& request, + const std::shared_ptr& http_request) const override; + + // Test seam for asserting the outgoing header without relying on ADC or metadata. + virtual std::optional fetch_token() const; + +private: + GcpTokenProvider _token_provider; +}; + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp b/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp new file mode 100644 index 00000000000000..596c4370dac4f9 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp @@ -0,0 +1,93 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/gcp/gcp_token_provider.h" + +#include +#include +#include + +#include + +namespace doris { +namespace { + +constexpr char GCS_READ_WRITE_SCOPE[] = "https://www.googleapis.com/auth/devstorage.read_write"; +constexpr char CLOUD_PLATFORM_SCOPE[] = "https://www.googleapis.com/auth/cloud-platform"; + +std::shared_ptr make_credentials(const GcpCredentialConfig& credential, + const std::string& ca_cert_path, + GcpTokenScope token_scope) { + google::cloud::Options options; + if (!ca_cert_path.empty()) { + options.set(ca_cert_path); + } + + // A source credential used for impersonation calls the IAM Credentials API + // and therefore needs cloud-platform. The resulting target token uses the + // caller-selected scope: object read/write for storage requests, or + // cloud-platform when it will call IAM signBlob. + const char* requested_scope = token_scope == GcpTokenScope::CloudPlatform + ? CLOUD_PLATFORM_SCOPE + : GCS_READ_WRITE_SCOPE; + options.set({credential.impersonation_service_account.empty() + ? requested_scope + : CLOUD_PLATFORM_SCOPE}); + + std::shared_ptr credentials; + switch (credential.provider_type) { + case GcpCredentialProviderType::Default: + credentials = google::cloud::MakeGoogleDefaultCredentials(options); + break; + case GcpCredentialProviderType::ComputeEngine: + credentials = google::cloud::MakeComputeEngineCredentials(options); + break; + } + if (!credential.impersonation_service_account.empty()) { + auto impersonation_options = options; + impersonation_options.set({requested_scope}); + credentials = google::cloud::MakeImpersonateServiceAccountCredentials( + std::move(credentials), credential.impersonation_service_account, + std::move(impersonation_options)); + } + return credentials; +} + +} // namespace + +GcpTokenProvider::GcpTokenProvider(const GcpCredentialConfig& credential, + const std::string& ca_cert_path, GcpTokenScope token_scope) + : _credentials(make_credentials(credential, ca_cert_path, token_scope)), + _token_generator(google::cloud::oauth2::MakeAccessTokenGenerator(*_credentials)) {} + +GcpTokenProvider::~GcpTokenProvider() = default; + +std::optional GcpTokenProvider::get_token() const { + // GcpS3Client and presign callers may share one provider across request + // threads. Serialize access even if the underlying implementation changes + // its thread-safety guarantees in a future google-cloud-cpp release. + std::lock_guard lock(_token_generator_mutex); + auto token = _token_generator->GetToken(); + if (!token) { + LOG_EVERY_N(WARNING, 100) << "failed to obtain GCP access token: " + << token.status().message(); + return std::nullopt; + } + return token->token; +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_token_provider.h b/common/cpp/obj-client/auth/gcp/gcp_token_provider.h new file mode 100644 index 00000000000000..8c39aa2b2dcb90 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcp_token_provider.h @@ -0,0 +1,57 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include + +#include +#include +#include +#include + +#include "cpp/obj-client/auth/gcp/gcp_auth.h" + +namespace doris { + +enum class GcpTokenScope { + StorageReadWrite, + CloudPlatform, +}; + +// Owns both objects because MakeAccessTokenGenerator takes Credentials by const +// reference. The Google auth library handles token caching and refresh internally. +class GcpTokenProvider { +public: + explicit GcpTokenProvider(const GcpCredentialConfig& credential, + const std::string& ca_cert_path = "", + GcpTokenScope token_scope = GcpTokenScope::StorageReadWrite); + ~GcpTokenProvider(); + + GcpTokenProvider(const GcpTokenProvider&) = delete; + GcpTokenProvider& operator=(const GcpTokenProvider&) = delete; + + std::optional get_token() const; + +private: + std::shared_ptr _credentials; + std::shared_ptr _token_generator; + mutable std::mutex _token_generator_mutex; +}; + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp b/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp new file mode 100644 index 00000000000000..9adbd09818ccf8 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp @@ -0,0 +1,194 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/gcp/gcs_signed_url.h" + +#include +#include + +#include +#include +#include +#include +#include + +#include "cpp/obj-client/auth/gcp/gcp_auth.h" + +namespace doris { +namespace { + +constexpr std::string_view SIGNING_ALGORITHM = "GOOG4-RSA-SHA256"; +constexpr std::string_view SIGNED_HEADERS = "host"; +constexpr std::string_view SIGNING_REGION = "auto"; +constexpr std::string_view SIGNING_SERVICE = "storage"; +constexpr std::string_view SIGNING_TERMINATOR = "goog4_request"; +constexpr int64_t MAX_EXPIRATION_SECONDS = 7 * 24 * 60 * 60; + +struct Endpoint { + std::string scheme; + std::string authority; +}; + +bool is_unreserved(unsigned char c) { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '-' || + c == '_' || c == '.' || c == '~'; +} + +std::string percent_encode(std::string_view value, bool preserve_slash) { + constexpr char HEX[] = "0123456789ABCDEF"; + std::string encoded; + encoded.reserve(value.size()); + for (unsigned char c : value) { + if (is_unreserved(c) || (preserve_slash && c == '/')) { + encoded.push_back(static_cast(c)); + continue; + } + encoded.push_back('%'); + encoded.push_back(HEX[c >> 4]); + encoded.push_back(HEX[c & 0x0F]); + } + return encoded; +} + +std::string hex_encode(std::string_view value) { + constexpr char HEX[] = "0123456789abcdef"; + std::string encoded; + encoded.reserve(value.size() * 2); + for (unsigned char c : value) { + encoded.push_back(HEX[c >> 4]); + encoded.push_back(HEX[c & 0x0F]); + } + return encoded; +} + +std::string sha256_hex(std::string_view value) { + std::array digest {}; + SHA256(reinterpret_cast(value.data()), value.size(), digest.data()); + return hex_encode( + std::string_view(reinterpret_cast(digest.data()), digest.size())); +} + +std::optional parse_endpoint(std::string endpoint, Endpoint* parsed) { + if (endpoint.empty()) { + return "GCS endpoint must not be empty"; + } + auto scheme_end = endpoint.find("://"); + if (scheme_end == std::string::npos) { + parsed->scheme = "https"; + } else { + parsed->scheme = endpoint.substr(0, scheme_end); + endpoint.erase(0, scheme_end + 3); + } + if (parsed->scheme != "http" && parsed->scheme != "https") { + return fmt::format("unsupported GCS endpoint scheme: {}", parsed->scheme); + } + while (endpoint.ends_with('/')) { + endpoint.pop_back(); + } + if (endpoint.empty() || endpoint.find_first_of("/?#") != std::string::npos) { + return fmt::format( + "GCS signed URL endpoint must not contain a path, query, " + "or fragment: {}", + endpoint); + } + parsed->authority = std::move(endpoint); + return std::nullopt; +} + +std::optional format_signing_time(std::chrono::system_clock::time_point now, + std::string* timestamp, std::string* date) { + auto time = std::chrono::system_clock::to_time_t(now); + std::tm utc {}; +#if defined(_WIN32) + if (gmtime_s(&utc, &time) != 0) { + return "failed to convert GCS signing time to UTC"; + } +#else + if (gmtime_r(&time, &utc) == nullptr) { + return "failed to convert GCS signing time to UTC"; + } +#endif + char timestamp_buffer[17]; + char date_buffer[9]; + if (std::strftime(timestamp_buffer, sizeof(timestamp_buffer), "%Y%m%dT%H%M%SZ", &utc) == 0 || + std::strftime(date_buffer, sizeof(date_buffer), "%Y%m%d", &utc) == 0) { + return "failed to format GCS signing time"; + } + *timestamp = timestamp_buffer; + *date = date_buffer; + return std::nullopt; +} + +} // namespace + +GcsV4SignedUrlResult build_gcs_v4_signed_url(const GcsV4SignedUrlOptions& options, + std::chrono::system_clock::time_point now, + const GcsSignBlobFunction& sign_blob) { + if (options.bucket.empty()) { + return {.error = "GCS bucket must not be empty"}; + } + if (!is_valid_gcp_service_account_email(options.signer_email)) { + return {.error = "invalid GCS signing service account email"}; + } + if (options.expiration_secs <= 0 || options.expiration_secs > MAX_EXPIRATION_SECONDS) { + return {.error = fmt::format("GCS signed URL expiration must be in [1, {}] seconds", + MAX_EXPIRATION_SECONDS)}; + } + if (!sign_blob) { + return {.error = "GCS signBlob callback must not be empty"}; + } + + Endpoint endpoint; + if (auto error = parse_endpoint(options.endpoint, &endpoint); error.has_value()) { + return {.error = std::move(*error)}; + } + std::string timestamp; + std::string date; + if (auto error = format_signing_time(now, ×tamp, &date); error.has_value()) { + return {.error = std::move(*error)}; + } + + const std::string canonical_uri = + "/" + percent_encode(options.bucket, false) + "/" + percent_encode(options.key, true); + const std::string credential_scope = + fmt::format("{}/{}/{}/{}", date, SIGNING_REGION, SIGNING_SERVICE, SIGNING_TERMINATOR); + const std::string credential = options.signer_email + "/" + credential_scope; + const std::string canonical_query = fmt::format( + "X-Goog-Algorithm={}&X-Goog-Credential={}&X-Goog-Date={}&" + "X-Goog-Expires={}&X-Goog-SignedHeaders={}", + SIGNING_ALGORITHM, percent_encode(credential, false), timestamp, + options.expiration_secs, SIGNED_HEADERS); + const std::string canonical_request = + fmt::format("GET\n{}\n{}\nhost:{}\n\n{}\nUNSIGNED-PAYLOAD", canonical_uri, + canonical_query, endpoint.authority, SIGNED_HEADERS); + const std::string string_to_sign = fmt::format("{}\n{}\n{}\n{}", SIGNING_ALGORITHM, timestamp, + credential_scope, sha256_hex(canonical_request)); + + auto sign_result = sign_blob(string_to_sign); + if (!sign_result.ok()) { + return {.error = std::move(sign_result.error)}; + } + if (sign_result.signature.empty()) { + return {.error = "signBlob returned an empty signature"}; + } + + return {.signed_url = fmt::format("{}://{}{}?{}&X-Goog-Signature={}", endpoint.scheme, + endpoint.authority, canonical_uri, canonical_query, + hex_encode(sign_result.signature))}; +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcs_signed_url.h b/common/cpp/obj-client/auth/gcp/gcs_signed_url.h new file mode 100644 index 00000000000000..8cc152925b4551 --- /dev/null +++ b/common/cpp/obj-client/auth/gcp/gcs_signed_url.h @@ -0,0 +1,62 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include +#include + +namespace doris { + +struct GcsV4SignedUrlOptions { + std::string endpoint; + std::string bucket; + std::string key; + // The transport layer resolves this identity before invoking the builder. + std::string signer_email; + int64_t expiration_secs = 0; +}; + +struct GcsSignBlobResult { + std::string signature {}; + std::string error {}; + + bool ok() const noexcept { return error.empty(); } +}; + +// Returns the raw signature bytes for the supplied GCS V4 string-to-sign. The +// callback intentionally contains no transport-specific type so this signing +// algorithm can be shared by BE and cloud targets. +using GcsSignBlobFunction = std::function; + +struct GcsV4SignedUrlResult { + std::string signed_url {}; + std::string error {}; + + bool ok() const noexcept { return error.empty(); } +}; + +// Builds a GCS V4 signed URL from a caller-supplied signing function. Resolving +// credentials and invoking IAM signBlob belong to the caller's transport layer. +GcsV4SignedUrlResult build_gcs_v4_signed_url(const GcsV4SignedUrlOptions& options, + std::chrono::system_clock::time_point now, + const GcsSignBlobFunction& sign_blob); + +} // namespace doris diff --git a/common/cpp/obj-client/auth/obj_credential.cpp b/common/cpp/obj-client/auth/obj_credential.cpp new file mode 100644 index 00000000000000..8317059e34e890 --- /dev/null +++ b/common/cpp/obj-client/auth/obj_credential.cpp @@ -0,0 +1,134 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/obj_credential.h" + +#include +#include +#include + +namespace doris { +namespace { + +struct CredentialParseResult { + CredentialConfig credential {}; + std::optional error = std::nullopt; +}; + +// Adapt provider-owned parsers to the generic registry. A provider parser +// returns no credential when none of its properties are present and an error +// for a partially configured credential. +CredentialParseResult parse_gcp_properties(const ObjCredentialPropertyGetter& get_property) { + auto result = parse_gcp_credential_properties(get_property); + if (result.error.has_value()) { + return {.error = std::move(result.error)}; + } + if (!result.credential.has_value()) { + return {}; + } + return {.credential = std::move(*result.credential)}; +} + +using CredentialPropertyParser = + CredentialParseResult (*)(const ObjCredentialPropertyGetter& get_property); + +// Register exactly one parser per non-monostate CredentialConfig alternative. +// The loop below also rejects properties from multiple credential providers. +// Add parse_oss_credential_properties/parse_azure_credential_properties here +// when those passwordless modes are introduced. +constexpr std::array CREDENTIAL_PROPERTY_PARSERS { + CredentialPropertyParser {parse_gcp_properties}, +}; + +static_assert(CREDENTIAL_PROPERTY_PARSERS.size() + 1 == std::variant_size_v); + +// These visitors are the runtime extension points. Do not add a catch-all +// template overload: explicit overloads make a new CredentialConfig type fail +// to compile until its hashing and validation are implemented. +struct CredentialHashVisitor { + uint64_t* hash; + + void operator()(const std::monostate&) const {} + + void operator()(const GcpCredentialConfig& credential) const { + *hash ^= static_cast(credential.provider_type) + 1; + *hash ^= std::hash {}(credential.impersonation_service_account); + } +}; + +struct CredentialValidationVisitor { + const ObjCredentialValidationContext& context; + + std::optional operator()(const std::monostate&) const { return std::nullopt; } + + std::optional operator()(const GcpCredentialConfig& credential) const { + if (context.provider != ObjCredentialProvider::Gcp) { + return "GCP credentials require provider=GCP"; + } + if (context.has_conflicting_credentials()) { + return "GCP credentials cannot be combined with other credentials"; + } + return validate_gcp_credential(credential); + } +}; + +struct CredentialTypeNameVisitor { + std::string_view operator()(const std::monostate&) const { return "none"; } + std::string_view operator()(const GcpCredentialConfig&) const { return "gcp"; } +}; + +} // namespace + +void hash_combine_credential(uint64_t* hash, const CredentialConfig& credential) { + *hash ^= credential.index(); + std::visit(CredentialHashVisitor {hash}, credential); +} + +std::string_view credential_type_name(const CredentialConfig& credential) { + return std::visit(CredentialTypeNameVisitor {}, credential); +} + +std::optional validate_obj_credential_config( + const CredentialConfig& credential, const ObjCredentialValidationContext& context) { + return std::visit(CredentialValidationVisitor {context}, credential); +} + +std::optional parse_obj_credential_properties( + const ObjCredentialPropertyGetter& get_property, CredentialConfig* credential) { + CredentialConfig selected; + + for (auto parser : CREDENTIAL_PROPERTY_PARSERS) { + auto result = parser(get_property); + if (result.error.has_value()) { + return std::move(result.error); + } + if (std::holds_alternative(result.credential)) { + continue; + } + if (!std::holds_alternative(selected)) { + return "multiple object storage credential types are configured"; + } + selected = std::move(result.credential); + } + + if (!std::holds_alternative(selected)) { + *credential = std::move(selected); + } + return std::nullopt; +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/obj_credential.h b/common/cpp/obj-client/auth/obj_credential.h new file mode 100644 index 00000000000000..6d32385588d5f8 --- /dev/null +++ b/common/cpp/obj-client/auth/obj_credential.h @@ -0,0 +1,70 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "cpp/obj-client/auth/gcp/gcp_auth.h" +#include "cpp/obj-client/auth/obj_credential_common.h" + +namespace doris { + +class TCredential; + +namespace cloud { +class ObjectStoreInfoPB; +} + +// This abstraction keeps provider-specific passwordless authentication out of +// S3 utilities and provides one runtime credential model shared by BE and Cloud. +// To add another provider (for example OSS or Azure): +// 1. add CredentialConfig in its own _auth.h; +// 2. add that type to this variant; and +// 3. implement the corresponding hash/validate visitors, client builder, and +// protocol/property conversions. +// The explicit visitors intentionally make an incomplete extension fail to compile. +using CredentialConfig = std::variant; + +void hash_combine_credential(uint64_t* hash, const CredentialConfig& credential); + +std::string_view credential_type_name(const CredentialConfig& credential); + +std::optional validate_obj_credential_config( + const CredentialConfig& credential, const ObjCredentialValidationContext& context); + +std::optional parse_obj_credential_properties( + const ObjCredentialPropertyGetter& get_property, CredentialConfig* credential); + +template +std::optional parse_obj_credential_properties( + const std::map& properties, + CredentialConfig* credential) { + return parse_obj_credential_properties(make_property_getter(properties), credential); +} + +void convert_obj_credential(const cloud::ObjectStoreInfoPB& source, CredentialConfig* credential); +void convert_obj_credential(const TCredential& source, CredentialConfig* credential); + +} // namespace doris diff --git a/common/cpp/obj-client/auth/obj_credential_common.h b/common/cpp/obj-client/auth/obj_credential_common.h new file mode 100644 index 00000000000000..0e855fe57f1354 --- /dev/null +++ b/common/cpp/obj-client/auth/obj_credential_common.h @@ -0,0 +1,88 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include + +namespace doris { + +// Credential-layer provider identity. Storage and protocol enums are converted +// to this type only at their respective validation boundaries. +enum class ObjCredentialProvider { + Unknown, + Aws, + Azure, + Bos, + Cos, + Gcp, + Obs, + Oss, + Tos, +}; + +// TODO: Add AwsCredentialConfig to CredentialConfig so AWS uses the same +// provider-neutral path. The AWS-specific role ARN, external ID, and credential- +// provider fields remain outside this abstraction for compatibility with existing +// configurations and protocol clients. AK/SK and session tokens are common static +// credentials for S3-compatible providers and are not part of this migration. +struct ObjCredentialValidationContext { + ObjCredentialProvider provider = ObjCredentialProvider::Unknown; + std::string_view ak {}; + std::string_view sk {}; + std::string_view token {}; + bool has_aws_role_arn = false; + bool has_aws_external_id = false; + bool has_aws_credential_provider = false; + bool has_encrypted_access_keys = false; + + bool has_static_credentials() const { + return !ak.empty() || !sk.empty() || !token.empty() || has_encrypted_access_keys; + } + + bool has_aws_authentication() const { + return has_aws_role_arn || has_aws_external_id || has_aws_credential_provider; + } + + // A provider-native credential is mutually exclusive with both common + // static credentials and provider-specific authentication fields. + bool has_conflicting_credentials() const { + return has_static_credentials() || has_aws_authentication(); + } +}; + +// Return an owned string so provider parsers never depend on a caller's storage +// or on the lifetime of a temporary normalized property value. +using ObjCredentialPropertyGetter = std::function(std::string_view)>; + +template +ObjCredentialPropertyGetter make_property_getter(const PropertyMap& properties) { + // Preserve the caller's comparator (including StringCaseMap's + // case-insensitive lookup) instead of normalizing property names here. + return [&properties](std::string_view key) -> std::optional { + auto it = properties.find(std::string(key)); + if (it == properties.end()) { + return std::nullopt; + } + return it->second; + }; +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/obj_s3_client_factory.cpp b/common/cpp/obj-client/auth/obj_s3_client_factory.cpp new file mode 100644 index 00000000000000..45ffa7e3bd4159 --- /dev/null +++ b/common/cpp/obj-client/auth/obj_s3_client_factory.cpp @@ -0,0 +1,54 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/obj_s3_client_factory.h" + +#include + +#include + +#include "cpp/obj-client/auth/gcp/gcp_s3_client.h" + +namespace doris { +namespace { + +// Keep the overloads explicit so adding another CredentialConfig alternative +// fails to compile until its client construction path is implemented here. +struct S3ClientBuilder { + S3ClientBuildContext& context; + + std::shared_ptr operator()(const std::monostate&) const { + return std::make_shared( + context.fallback_provider, std::move(context.config), + context.payload_signing_policy, context.use_virtual_addressing); + } + + std::shared_ptr operator()(const GcpCredentialConfig& credential) const { + return std::make_shared( + credential, context.ca_cert_path, std::move(context.config), + context.payload_signing_policy, context.use_virtual_addressing); + } +}; + +} // namespace + +std::shared_ptr make_s3_client(const CredentialConfig& credential, + S3ClientBuildContext context) { + return std::visit(S3ClientBuilder {context}, credential); +} + +} // namespace doris diff --git a/common/cpp/obj-client/auth/obj_s3_client_factory.h b/common/cpp/obj-client/auth/obj_s3_client_factory.h new file mode 100644 index 00000000000000..bd4d104de958cb --- /dev/null +++ b/common/cpp/obj-client/auth/obj_s3_client_factory.h @@ -0,0 +1,50 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include + +#include +#include + +#include "cpp/obj-client/auth/obj_credential.h" + +namespace Aws::S3 { +class S3Client; +} + +namespace doris { + +// Neutral inputs shared by BE's S3ClientConf and Cloud's S3Conf. Provider-native +// authentication is selected before construction because AWS credentials providers +// cannot be replaced safely on an already constructed client. +struct S3ClientBuildContext { + std::shared_ptr fallback_provider; + Aws::Client::ClientConfiguration config; + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy payload_signing_policy = + Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never; + bool use_virtual_addressing = true; + std::string ca_cert_path; +}; + +std::shared_ptr make_s3_client(const CredentialConfig& credential, + S3ClientBuildContext context); + +} // namespace doris diff --git a/common/cpp/obj-client/s3_obj_storage_client.cpp b/common/cpp/obj-client/s3_obj_storage_client.cpp index df9210f612ee38..ceea3cfea6a0c8 100644 --- a/common/cpp/obj-client/s3_obj_storage_client.cpp +++ b/common/cpp/obj-client/s3_obj_storage_client.cpp @@ -501,6 +501,9 @@ ObjStorageResponse S3ObjStorageClient::delete_object(const ObjStoragePath& opts) std::string S3ObjStorageClient::generate_presigned_url(const ObjStoragePath& opts, int64_t expiration_secs) { + if (_signed_url_generator) { + return _signed_url_generator(opts, expiration_secs); + } return _client->GeneratePresignedUrl(opts.bucket, opts.key, Aws::Http::HttpMethod::HTTP_GET, expiration_secs); } diff --git a/common/cpp/obj-client/s3_obj_storage_client.h b/common/cpp/obj-client/s3_obj_storage_client.h index 1f18bde7431ba1..dc45530fe4ddb0 100644 --- a/common/cpp/obj-client/s3_obj_storage_client.h +++ b/common/cpp/obj-client/s3_obj_storage_client.h @@ -67,6 +67,7 @@ #include #include +#include #include #include @@ -89,9 +90,14 @@ ObjStorageStatus s3fs_error(const Aws::S3::S3Error& err, std::string_view msg); class S3ObjStorageClient final : public ObjStorageClient { public: + using SignedUrlGenerator = std::function; + S3ObjStorageClient(std::shared_ptr client, - ObjStorageEndpointInfo config = {}) - : _config(std::move(config)), _client(std::move(client)) {} + ObjStorageEndpointInfo config = {}, + SignedUrlGenerator signed_url_generator = {}) + : _config(std::move(config)), + _client(std::move(client)), + _signed_url_generator(std::move(signed_url_generator)) {} ~S3ObjStorageClient() override = default; ObjStorageUploadResult create_multipart_upload(const ObjStoragePath& opts) override; ObjStorageResponse put_object(const ObjStoragePath& opts, std::string_view stream) override; @@ -125,6 +131,7 @@ class S3ObjStorageClient final : public ObjStorageClient { private: ObjStorageEndpointInfo _config; std::shared_ptr _client; + SignedUrlGenerator _signed_url_generator; }; } // namespace doris diff --git a/fe/be-java-extensions/hadoop-deps/pom.xml b/fe/be-java-extensions/hadoop-deps/pom.xml index 0f7c94cf621eca..ef8b3a89a66f87 100644 --- a/fe/be-java-extensions/hadoop-deps/pom.xml +++ b/fe/be-java-extensions/hadoop-deps/pom.xml @@ -73,6 +73,11 @@ under the License. org.apache.hadoop hadoop-common + + com.google.cloud.bigdataoss + gcs-connector + shaded + org.apache.hadoop hadoop-azure diff --git a/fe/fe-connector/fe-connector-iceberg/pom.xml b/fe/fe-connector/fe-connector-iceberg/pom.xml index 856594ff29f2a5..922dc0a7fa3afa 100644 --- a/fe/fe-connector/fe-connector-iceberg/pom.xml +++ b/fe/fe-connector/fe-connector-iceberg/pom.xml @@ -41,6 +41,18 @@ under the License. + + + org.apache.doris + fe-filesystem-gcs + ${revision} + + + + com.google.cloud.bigdataoss + gcs-connector + shaded + ${project.groupId} fe-connector-spi diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java new file mode 100644 index 00000000000000..fac1530c53db95 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.filesystem.gcs.GcpAuth; +import org.apache.doris.filesystem.gcs.GcsFileSystemProperties; + +import org.apache.iceberg.aws.s3.S3FileIOAwsClientFactory; +import software.amazon.awssdk.services.s3.S3AsyncClient; +import software.amazon.awssdk.services.s3.S3Client; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.util.HashMap; +import java.util.Map; + +/** Supplies Iceberg S3FileIO clients configured with native GCP bearer authentication. */ +public class GcpS3FileIOAwsClientFactory implements S3FileIOAwsClientFactory { + private static final long serialVersionUID = 1L; + + private Map properties; + + public GcpS3FileIOAwsClientFactory() { + } + + @Override + public void initialize(Map properties) { + this.properties = new HashMap<>(properties); + } + + @Override + public S3Client s3() { + try { + return GcpAuth.createS3Client(gcsProperties()); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + @Override + public S3AsyncClient s3Async() { + try { + return GcpAuth.createS3AsyncClient(gcsProperties()); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + private GcsFileSystemProperties gcsProperties() { + if (properties == null) { + throw new IllegalStateException("GCP Iceberg client factory has not been initialized"); + } + GcsFileSystemProperties storageProperties = GcsFileSystemProperties.of(properties); + if (storageProperties.getAuth().getNativeCredential().isEmpty()) { + throw new IllegalArgumentException("Native GCP Iceberg authentication requires GCS credentials"); + } + return storageProperties; + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index 054bfb76ea8552..49113697c77d87 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -22,6 +22,9 @@ import org.apache.doris.connector.metastore.iceberg.jdbc.IcebergJdbcMetaStoreProperties; import org.apache.doris.connector.metastore.iceberg.rest.IcebergRestMetaStoreProperties; import org.apache.doris.connector.spi.DorisConnectorException; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.GcsAuth; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.filesystem.properties.StorageProperties; @@ -419,6 +422,22 @@ public static Map buildCatalogProperties(IcebergCatalogPropertie // s3tables: bespoke instantiation. Preserve the skeleton's base+impl routing. break; } + chosenS3.ifPresent(storage -> storage.toBackendProperties().ifPresent(backend -> { + Optional auth = GcsAuthResolver.resolve(backend.toMap()); + if (auth.filter(GcsAuth::isAnonymous).isPresent()) { + opts.put(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER, + "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider"); + } + auth.flatMap(GcsAuth::getNativeCredential).ifPresent(credential -> { + putS3FileIODialect(opts, storage); + opts.put("provider", "GCP"); + opts.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); + putIfNotBlank(opts, GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, + credential.getImpersonationServiceAccount()); + opts.put(CatalogProperties.FILE_IO_IMPL, "org.apache.iceberg.aws.s3.S3FileIO"); + opts.put(S3FileIOProperties.CLIENT_FACTORY, GcpS3FileIOAwsClientFactory.class.getName()); + }); + })); // The iceberg SDK forbids both "type" and "catalog-impl"; legacy buildIcebergCatalog removes "type". opts.remove(CatalogUtil.ICEBERG_CATALOG_TYPE); return opts; diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index 26921878e6ba16..100305cdc6a400 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.metastore.iceberg.jdbc.IcebergJdbcMetaStoreProperties; import org.apache.doris.connector.metastore.iceberg.rest.IcebergRestMetaStoreProperties; import org.apache.doris.connector.spi.DorisConnectorException; +import org.apache.doris.filesystem.gcs.GcsFileSystemProperties; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.filesystem.properties.StorageProperties; @@ -1040,4 +1041,45 @@ public void assembleHiveConfFailsLoudOnMissingFile(@TempDir java.nio.file.Path t } } + @Test + void nativeGcpCatalogUsesBearerFileIoAndNativeHadoopConfiguration() { + for (String source : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + Map raw = Map.of("provider", "GCP", "warehouse", "gs://bucket/warehouse", + "gs.endpoint", "https://storage.googleapis.com", + "gs.credential_provider_type", source, + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"); + GcsFileSystemProperties storage = GcsFileSystemProperties.of(raw); + for (String catalogType : List.of("hadoop", "rest", "jdbc", "hms")) { + Map properties = new HashMap<>(raw); + properties.put("iceberg.catalog.type", catalogType); + properties.put("uri", "jdbc".equals(catalogType) ? "jdbc:postgresql://localhost/iceberg" + : "hms".equals(catalogType) ? "thrift://localhost:9083" : "http://localhost:8181"); + properties.put("jdbc.user", "user"); + properties.put("jdbc.password", "password"); + properties.put("iceberg.jdbc.uri", "jdbc:postgresql://localhost/iceberg"); + Map options = IcebergCatalogFactory.buildCatalogProperties( + IcebergCatalogProperties.of(properties), Optional.of(storage)); + Assertions.assertEquals("org.apache.iceberg.aws.s3.S3FileIO", options.get("io-impl")); + Assertions.assertEquals(GcpS3FileIOAwsClientFactory.class.getName(), options.get("s3.client-factory")); + Assertions.assertEquals(source, options.get("gs.credential_provider_type")); + Assertions.assertEquals("https://storage.googleapis.com", options.get("s3.endpoint")); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", + options.get("gs.impersonation_service_account")); + } + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + storage.toHadoopConfigurationMap().get("fs.gs.impl")); + } + } + + @Test + void anonymousGcpCatalogUsesUnsignedFileIo() { + Map raw = Map.of("provider", "GCP", "warehouse", "gs://bucket/warehouse", + "iceberg.catalog.type", "hadoop", "gs.credential_provider_type", "ANONYMOUS"); + Map options = IcebergCatalogFactory.buildCatalogProperties( + IcebergCatalogProperties.of(raw), + Optional.of(GcsFileSystemProperties.of(raw))); + Assertions.assertEquals(AnonymousCredentialsProvider.class.getName(), + options.get(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER)); + Assertions.assertFalse(options.containsKey("s3.client-factory")); + } } diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index 656b9847c76147..e0ac341c64511e 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -457,6 +457,11 @@ under the License. commons-codec commons-codec + + com.google.cloud.bigdataoss + gcs-connector + shaded + org.apache.hadoop hadoop-azure diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java index dad70399e5fe7f..025d3f07d4814b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BrokerDesc.java @@ -121,6 +121,20 @@ public String getFileLocation(String location) throws UserException { return (null != storageAdapter) ? storageAdapter.validateAndNormalizeUri(location) : location; } + /** + * Normalize EXPORT and OUTFILE destinations for the BE writer. + */ + public String getFileLocationForExport(String location) throws UserException { + // BE parses HTTP(S) paths as path-style, while FE defaults to virtual-hosted parsing. + // Preserve these paths and the existing path handling for non-S3 backends. + if (StorageType.S3.equals(getStorageType()) + && !StringUtils.startsWithIgnoreCase(location, "http://") + && !StringUtils.startsWithIgnoreCase(location, "https://")) { + return getFileLocation(location); + } + return location; + } + public static BrokerDesc createForStreamLoad() { return new BrokerDesc("", StorageType.STREAM, null); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java index dce783c0fc9e01..55b20d5188f4ec 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java @@ -582,6 +582,11 @@ private void analyzeBrokerDesc(Map copiedProps) throws UserExcep } String brokerName = copiedProps.get(PROP_BROKER_NAME); brokerDesc = new BrokerDesc(brokerName, copiedProps); + try { + filePath = brokerDesc.getFileLocationForExport(filePath); + } catch (RuntimeException e) { + throw new AnalysisException("Invalid outfile path: " + filePath + ", " + e.getMessage(), e); + } /* * Note on HDFS export behavior and URI handling: * diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index 0c70cd67ce4a26..8e0ddea6ef2c19 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -24,6 +24,7 @@ import org.apache.doris.common.util.S3Util; import org.apache.doris.datasource.storage.S3ResourceCompat; import org.apache.doris.filesystem.UploadPartResult; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.filesystem.spi.ObjStorage; import org.apache.doris.filesystem.spi.RequestBody; @@ -35,6 +36,7 @@ import com.google.common.collect.Lists; import com.google.common.collect.Maps; import com.google.gson.annotations.SerializedName; +import org.apache.commons.lang3.StringUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -44,7 +46,9 @@ import java.util.Collections; import java.util.HashMap; import java.util.List; +import java.util.Locale; import java.util.Map; +import java.util.Objects; import java.util.Optional; /** @@ -74,6 +78,20 @@ */ public class S3Resource extends Resource { private static final Logger LOG = LogManager.getLogger(S3Resource.class); + // Only aliases already supported by GCSProperties belong here. Resource/Vault options + // without a GCS alias (bucket, root path, region, validity checking) keep their existing names. + private static final Map GCS_PROPERTY_ALIASES = ImmutableMap.builder() + .put("gs.endpoint", S3ResourceCompat.ENDPOINT) + .put("gs.access_key", S3ResourceCompat.ACCESS_KEY) + .put("gs.secret_key", S3ResourceCompat.SECRET_KEY) + .put("gs.session_token", S3ResourceCompat.SESSION_TOKEN) + .put("gs.connection.maximum", S3ResourceCompat.MAX_CONNECTIONS) + .put("gs.connection.request.timeout", S3ResourceCompat.REQUEST_TIMEOUT_MS) + .put("gs.connection.timeout", S3ResourceCompat.CONNECTION_TIMEOUT_MS) + .put("gs.use_path_style", S3ResourceCompat.USE_PATH_STYLE) + .put("gs.force_parsing_by_standard_uri", "force_parsing_by_standard_uri") + .build(); + @SerializedName(value = "properties") private Map properties; @@ -94,6 +112,7 @@ public String getProperty(String propertyKey) { protected void setProperties(ImmutableMap newProperties) throws DdlException { Preconditions.checkState(newProperties != null); this.properties = Maps.newHashMap(newProperties); + normalizeProperties(this.properties, this.properties.get("provider")); // check properties S3ResourceCompat.requiredS3PingProperties(properties); @@ -218,72 +237,112 @@ protected static void pingS3(String bucketName, String rootPath, Map properties, String provider) { + // Validate the raw aliases before a preferred value can hide conflicting credentials. + GcsAuthResolver.resolve(properties); + if (provider == null) { + return; + } + switch (provider.toUpperCase(Locale.ROOT)) { + case "GCP": + // Normalize before validation, policy checks and persistence, so FE connector + // binding and Resource/Vault protocol builders consume the same values. + GCS_PROPERTY_ALIASES.forEach((alias, key) -> { + String value = properties.remove(alias); + // Match connector binding: nonblank gs.* values take precedence. + if (StringUtils.isNotBlank(value)) { + properties.put(key, value); + } + }); + break; + default: + break; + } + } + @Override - public void modifyProperties(Map properties) throws DdlException { + public synchronized void modifyProperties(Map newProperties) throws DdlException { + // Serialize the snapshot, validation and publication. A lock only around publication + // would allow a concurrent ALTER to replace a successful update with an older snapshot. + Map properties = new HashMap<>(newProperties); + String provider = StringUtils.defaultIfEmpty(properties.get("provider"), + this.properties.get("provider")); + // Preserve AWS_* ALTER compatibility before merging with stored canonical properties. + S3ResourceCompat.convertToStdProperties(properties); + Map effectiveProperties = new HashMap<>(this.properties); + for (Map.Entry update : properties.entrySet()) { + // Match persistence: empty updates are ignored, except when clearing a session token. + replaceIfEffectiveValue(effectiveProperties, update.getKey(), update.getValue()); + if (S3ResourceCompat.SESSION_TOKEN.equals(update.getKey()) + || S3ResourceCompat.Env.TOKEN.equals(update.getKey())) { + effectiveProperties.put(update.getKey(), update.getValue()); + } + } + // Resolve aliases only after merging: nonblank gs.* values, including stored aliases, + // take precedence over s3.* values in the final configuration. + normalizeProperties(effectiveProperties, provider); + S3ResourceCompat.convertToStdProperties(effectiveProperties); + + // Normalize the update keys for modification restrictions and credential-mode changes. + // Their values must not overwrite the configuration resolved above. + normalizeProperties(properties, provider); if (references.containsValue(ReferenceType.POLICY)) { // can't change, because remote fs use it info to find data. List cantChangeProperties = Arrays.asList(S3ResourceCompat.ENDPOINT, S3ResourceCompat.REGION, S3ResourceCompat.ROOT_PATH, S3ResourceCompat.BUCKET, S3ResourceCompat.Env.ENDPOINT, S3ResourceCompat.Env.REGION, S3ResourceCompat.Env.ROOT_PATH, S3ResourceCompat.Env.BUCKET); - Optional any = cantChangeProperties.stream().filter(properties::containsKey).findAny(); + Optional any = cantChangeProperties.stream() + .filter(key -> properties.containsKey(key) + || !Objects.equals(this.properties.get(key), effectiveProperties.get(key))) + .findAny(); if (any.isPresent()) { throw new DdlException("current not support modify property : " + any.get()); } } - // compatible with old version, Need convert if modified properties map uses old properties. - S3ResourceCompat.convertToStdProperties(properties); - if (!Strings.isNullOrEmpty(properties.get(S3ResourceCompat.ENDPOINT))) { - properties.put(S3ResourceCompat.Env.ENDPOINT, properties.get(S3ResourceCompat.ENDPOINT)); + if (!Strings.isNullOrEmpty(effectiveProperties.get(S3ResourceCompat.ENDPOINT))) { + effectiveProperties.put(S3ResourceCompat.Env.ENDPOINT, effectiveProperties.get(S3ResourceCompat.ENDPOINT)); } - boolean needCheck = isNeedCheck(properties); + for (Map.Entry kv : properties.entrySet()) { + if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ROLE_ARN) + && !Strings.isNullOrEmpty(kv.getValue())) { + effectiveProperties.remove(S3ResourceCompat.ACCESS_KEY); + effectiveProperties.remove(S3ResourceCompat.Env.ACCESS_KEY); + effectiveProperties.remove(S3ResourceCompat.SECRET_KEY); + effectiveProperties.remove(S3ResourceCompat.Env.SECRET_KEY); + } + if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ACCESS_KEY) + && !Strings.isNullOrEmpty(kv.getValue())) { + effectiveProperties.remove(S3ResourceCompat.ROLE_ARN); + effectiveProperties.remove(S3ResourceCompat.Env.ROLE_ARN); + effectiveProperties.remove(S3ResourceCompat.EXTERNAL_ID); + effectiveProperties.remove(S3ResourceCompat.Env.EXTERNAL_ID); + } + } + GcsAuthResolver.resolve(effectiveProperties); + boolean needCheck = isNeedCheck(effectiveProperties); if (LOG.isDebugEnabled()) { LOG.debug("s3 info need check validity : {}", needCheck); } if (needCheck) { - S3ResourceCompat.requiredS3PingProperties(this.properties); - Map changedProperties = new HashMap<>(this.properties); - changedProperties.putAll(properties); + S3ResourceCompat.requiredS3PingProperties(effectiveProperties); + Map changedProperties = new HashMap<>(effectiveProperties); String endpoint = S3Util.buildEndpointUrl(changedProperties.get(S3ResourceCompat.ENDPOINT)); changedProperties.put(S3ResourceCompat.ENDPOINT, endpoint); changedProperties.put(S3ResourceCompat.Env.ENDPOINT, endpoint); - String bucketName = properties.getOrDefault(S3ResourceCompat.BUCKET, - this.properties.get(S3ResourceCompat.BUCKET)); - String rootPath = properties.getOrDefault(S3ResourceCompat.ROOT_PATH, - this.properties.get(S3ResourceCompat.ROOT_PATH)); - - pingS3(bucketName, rootPath, changedProperties); + pingS3(effectiveProperties.get(S3ResourceCompat.BUCKET), + effectiveProperties.get(S3ResourceCompat.ROOT_PATH), + changedProperties); } - // modify properties writeLock(); - - for (Map.Entry kv : properties.entrySet()) { - replaceIfEffectiveValue(this.properties, kv.getKey(), kv.getValue()); - if (kv.getKey().equals(S3ResourceCompat.Env.TOKEN) - || kv.getKey().equals(S3ResourceCompat.SESSION_TOKEN)) { - this.properties.put(kv.getKey(), kv.getValue()); - } - - if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ROLE_ARN) - && !Strings.isNullOrEmpty(kv.getValue())) { - this.properties.remove(S3ResourceCompat.ACCESS_KEY); - this.properties.remove(S3ResourceCompat.Env.ACCESS_KEY); - this.properties.remove(S3ResourceCompat.SECRET_KEY); - this.properties.remove(S3ResourceCompat.Env.SECRET_KEY); - } - - if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ACCESS_KEY) - && !Strings.isNullOrEmpty(kv.getValue())) { - this.properties.remove(S3ResourceCompat.ROLE_ARN); - this.properties.remove(S3ResourceCompat.Env.ROLE_ARN); - this.properties.remove(S3ResourceCompat.EXTERNAL_ID); - this.properties.remove(S3ResourceCompat.Env.EXTERNAL_ID); - } + try { + this.properties = effectiveProperties; + ++version; + } finally { + writeUnlock(); } - ++version; - writeUnlock(); - super.modifyProperties(properties); + super.modifyProperties(effectiveProperties); } private CloudCredentialWithEndpoint getS3PingCredentials(Map properties) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java index 1546a2b687f23e..60ae2c88b56155 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java @@ -71,6 +71,8 @@ public static class PropertyKey { public static final String ROLE_ARN = "s3.role_arn"; public static final String EXTERNAL_ID = "s3.external_id"; public static final String CREDENTIALS_PROVIDER_TYPE = "s3.credentials_provider_type"; + public static final String GCP_CREDENTIAL_PROVIDER_TYPE = "gs.credential_provider_type"; + public static final String GCP_IMPERSONATION_SERVICE_ACCOUNT = "gs.impersonation_service_account"; } public static final HashSet ALLOW_ALTER_PROPERTIES = new HashSet<>(Arrays.asList( @@ -81,7 +83,9 @@ public static class PropertyKey { PropertyKey.USE_PATH_STYLE, PropertyKey.ROLE_ARN, PropertyKey.EXTERNAL_ID, - PropertyKey.CREDENTIALS_PROVIDER_TYPE + PropertyKey.CREDENTIALS_PROVIDER_TYPE, + PropertyKey.GCP_CREDENTIAL_PROVIDER_TYPE, + PropertyKey.GCP_IMPERSONATION_SERVICE_ACCOUNT )); @SerializedName(value = "properties") diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java index 4662c72d7ff621..643bdd6f1d1f2f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java @@ -26,6 +26,9 @@ import org.apache.doris.common.UserException; import org.apache.doris.common.lock.MonitoredReentrantReadWriteLock; import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; +import org.apache.doris.datasource.storage.S3ResourceCompat; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.proto.InternalService.PAlterVaultSyncRequest; import org.apache.doris.rpc.BackendServiceProxy; @@ -145,16 +148,57 @@ private void updateDefaultStorageVaultCache(Pair newDefaultVault } } - private Cloud.StorageVaultPB.Builder buildAlterS3VaultRequest(Map properties, String name) + private Cloud.StorageVaultPB.Builder buildS3VaultRequest(Map properties, String name) throws Exception { - Cloud.ObjectStoreInfoPB.Builder objBuilder = CloudObjectStoreAdapter.getObjStoreInfoPB(properties); - Cloud.StorageVaultPB.Builder alterObjVaultBuilder = Cloud.StorageVaultPB.newBuilder(); - alterObjVaultBuilder.setName(name); - alterObjVaultBuilder.setObjInfo(objBuilder.build()); + Cloud.StorageVaultPB.Builder builder = Cloud.StorageVaultPB.newBuilder() + .setName(name).setObjInfo(CloudObjectStoreAdapter.getObjStoreInfoPB(properties)); if (properties.containsKey(StorageVault.PropertyKey.VAULT_NAME)) { - alterObjVaultBuilder.setAlterName(properties.get(StorageVault.PropertyKey.VAULT_NAME)); + builder.setAlterName(properties.get(StorageVault.PropertyKey.VAULT_NAME)); } - return alterObjVaultBuilder; + return builder; + } + + private Cloud.StorageVaultPB.Builder buildAlterS3VaultRequest(Map properties, String name) + throws Exception { + Cloud.StorageVaultPB.Builder builder = buildS3VaultRequest(properties, name); + Cloud.ObjectStoreInfoPB.Builder objBuilder = builder.getObjInfoBuilder(); + // CREATE may infer GCP from native credentials. ALTER must not send an inferred + // provider because MS rejects changes to this immutable storage field. + if (!properties.containsKey("provider")) { + objBuilder.clearProvider(); + } + // getObjStoreInfoPB supplies INSTANCE_PROFILE as the create-time default + // when role_arn is present. ALTER is a patch operation, so only send the + // credential provider when the user explicitly changed it. + if (!properties.containsKey(S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE) + && !properties.containsKey(S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE)) { + objBuilder.clearCredProviderType(); + } + // Empty strings are meaningful for ALTER: they explicitly clear the + // stored role or external ID. The generic create builder omits them. + if (properties.containsKey(S3ResourceCompat.ROLE_ARN)) { + objBuilder.setRoleArn(properties.get(S3ResourceCompat.ROLE_ARN)); + } + if (properties.containsKey(S3ResourceCompat.EXTERNAL_ID)) { + objBuilder.setExternalId(properties.get(S3ResourceCompat.EXTERNAL_ID)); + } + // GCS CREATE defaults to ADC, but a storage-only ALTER must not change authentication. + if (!GcsAuthResolver.hasNativeCredentialProperties(properties) + && objBuilder.hasCredential() && objBuilder.getCredential().hasGcpCredential()) { + objBuilder.clearCredential(); + } + if (objBuilder.hasCredential() && objBuilder.getCredential().hasGcpCredential()) { + Cloud.GcpCredentialPB.Builder credential = objBuilder.getCredentialBuilder().getGcpCredentialBuilder(); + if (!properties.containsKey(GcpCredential.CREDENTIAL_PROVIDER_TYPE)) { + credential.clearCredentialProviderType(); + } + // Preserve explicit empty strings so MS can distinguish clearing + // impersonation from leaving the stored target unchanged. + if (properties.containsKey(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)) { + credential.setImpersonationServiceAccount(properties.get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + } + } + return builder; } private Cloud.StorageVaultPB.Builder buildAlterHdfsVaultRequest(Map properties, String name) @@ -182,9 +226,15 @@ private Cloud.StorageVaultPB.Builder buildAlterStorageVaultRequest(StorageVaultT return builder; } - private Cloud.StorageVaultPB.Builder buildAlterStorageVaultRequest(StorageVault vault) throws Exception { - Cloud.StorageVaultPB.Builder builder = buildAlterStorageVaultRequest(vault.getType(), - vault.getCopiedProperties(), vault.getName()); + private Cloud.StorageVaultPB.Builder buildCreateStorageVaultRequest(StorageVault vault) throws Exception { + Cloud.StorageVaultPB.Builder builder; + if (vault.getType() == StorageVaultType.S3) { + builder = buildS3VaultRequest(vault.getCopiedProperties(), vault.getName()); + } else if (vault.getType() == StorageVaultType.HDFS) { + builder = buildAlterHdfsVaultRequest(vault.getCopiedProperties(), vault.getName()); + } else { + throw new DdlException("Unknown storage vault type"); + } Cloud.StorageVaultPB.PathFormat.Builder pathBuilder = Cloud.StorageVaultPB.PathFormat.newBuilder(); pathBuilder.setShardNum(vault.getNumShard()); pathBuilder.setPathVersion(vault.getPathVersion()); @@ -209,8 +259,8 @@ public void alterStorageVault(StorageVaultType type, Map propert request.setOp(Operation.ALTER_S3_VAULT); } else if (type == StorageVaultType.HDFS) { properties.keySet().stream() - // Exact literals of the legacy S3Properties.S3_PREFIX ("s3.") and - // StorageProperties.FS_PROVIDER_KEY ("provider") constants. + // Exact literals of the legacy S3ResourceCompat.S3_PREFIX ("s3.") and + // "provider" ("provider") constants. .filter(key -> HdfsStorageVault.FORBID_ALTER_PROPERTIES.contains(key) || key.toLowerCase().contains("s3.") || key.toLowerCase().contains("provider")) @@ -321,7 +371,7 @@ public StorageVaultType getStorageVaultTypeByName(String vaultName) throws DdlEx @VisibleForTesting public void createHdfsVault(StorageVault vault) throws Exception { - Cloud.StorageVaultPB.Builder alterHdfsInfoBuilder = buildAlterStorageVaultRequest(vault); + Cloud.StorageVaultPB.Builder alterHdfsInfoBuilder = buildCreateStorageVaultRequest(vault); Cloud.AlterObjStoreInfoRequest.Builder requestBuilder = Cloud.AlterObjStoreInfoRequest.newBuilder().setRequestIp(FrontendOptions.getLocalHostAddressCached()); requestBuilder.setOp(Cloud.AlterObjStoreInfoRequest.Operation.ADD_HDFS_INFO); @@ -371,7 +421,7 @@ private void alterSyncVaultTask() { } public void createS3Vault(StorageVault vault) throws Exception { - Cloud.StorageVaultPB.Builder s3StorageVaultBuilder = buildAlterStorageVaultRequest(vault); + Cloud.StorageVaultPB.Builder s3StorageVaultBuilder = buildCreateStorageVaultRequest(vault); Cloud.AlterObjStoreInfoRequest.Builder requestBuilder = Cloud.AlterObjStoreInfoRequest.newBuilder().setRequestIp(FrontendOptions.getLocalHostAddressCached()); requestBuilder.setOp(Cloud.AlterObjStoreInfoRequest.Operation.ADD_S3_VAULT); diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java index dc3959f01d1624..7f9d84ec743722 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java +++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java @@ -38,6 +38,7 @@ public class ObjectInfo { // only used for aws private final String externalId; private final String token; + private final Cloud.GcpCredentialPB gcpCredential; // Used to get sts token public ObjectInfo(Cloud.ObjectStoreInfoPB.Provider provider, String ak, String sk, @@ -56,19 +57,26 @@ public ObjectInfo(Cloud.ObjectStoreInfoPB objectStoreInfoPB) { this(objectStoreInfoPB.getProvider(), objectStoreInfoPB.getAk(), objectStoreInfoPB.getSk(), objectStoreInfoPB.getBucket(), objectStoreInfoPB.getEndpoint(), objectStoreInfoPB.getRegion(), objectStoreInfoPB.getPrefix(), null, objectStoreInfoPB.getRoleArn(), - objectStoreInfoPB.getExternalId(), null); + objectStoreInfoPB.getExternalId(), null, nativeCredential(objectStoreInfoPB)); } public ObjectInfo(Cloud.ObjectStoreInfoPB objectStoreInfoPB, String roleName, String arn, String externalId, String token) { this(objectStoreInfoPB.getProvider(), objectStoreInfoPB.getAk(), objectStoreInfoPB.getSk(), objectStoreInfoPB.getBucket(), objectStoreInfoPB.getEndpoint(), objectStoreInfoPB.getRegion(), - objectStoreInfoPB.getPrefix(), roleName, arn, externalId, token); + objectStoreInfoPB.getPrefix(), roleName, arn, externalId, token, nativeCredential(objectStoreInfoPB)); } public ObjectInfo(Cloud.ObjectStoreInfoPB.Provider provider, String ak, String sk, String bucket, String endpoint, String region, String prefix, String roleName, String arn, String externalId, String token) { + this(provider, ak, sk, bucket, endpoint, region, prefix, roleName, arn, externalId, token, null); + } + + private ObjectInfo(Cloud.ObjectStoreInfoPB.Provider provider, String ak, String sk, String bucket, + String endpoint, String region, String prefix, String roleName, String arn, String externalId, + String token, Cloud.GcpCredentialPB gcpCredential) { + this.gcpCredential = gcpCredential; this.provider = provider; this.ak = ak; this.sk = sk; @@ -86,6 +94,15 @@ public Cloud.ObjectStoreInfoPB.Provider getProvider() { return provider; } + private static Cloud.GcpCredentialPB nativeCredential(Cloud.ObjectStoreInfoPB info) { + return info.hasCredential() && info.getCredential().hasGcpCredential() + ? info.getCredential().getGcpCredential() : null; + } + + public Cloud.GcpCredentialPB getGcpCredential() { + return gcpCredential; + } + public String getAk() { return ak; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java index e38aacfa6a6ba5..3f565af192c397 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java @@ -74,8 +74,17 @@ public static StorageAdapter toStorageAdapter(ObjectInfo obj) { case OSS: return StorageAdapter.ofProvider("OSS", buildS3CompatibleProps(obj)); case S3: - case GCP: return StorageAdapter.ofProvider("S3", buildS3CompatibleProps(obj)); + case GCP: + Map gcpProperties = buildS3CompatibleProps(obj); + gcpProperties.put("provider", "GCP"); + if (obj.getGcpCredential() != null) { + gcpProperties.put("gs.credential_provider_type", + obj.getGcpCredential().getCredentialProviderType().name()); + putIfNotBlank(gcpProperties, "gs.impersonation_service_account", + obj.getGcpCredential().getImpersonationServiceAccount()); + } + return StorageAdapter.ofProvider("GCS", gcpProperties); case COS: return StorageAdapter.ofProvider("COS", buildS3CompatibleProps(obj)); case OBS: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java new file mode 100644 index 00000000000000..073529602235c0 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java @@ -0,0 +1,71 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.property.storage.auth; + +import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.GcpCredentialProviderType; +import org.apache.doris.thrift.TCredential; +import org.apache.doris.thrift.TGcpCredential; +import org.apache.doris.thrift.TGcpCredentialProviderType; +import org.apache.doris.thrift.TObjStorageType; +import org.apache.doris.thrift.TS3StorageParam; + +import org.apache.commons.lang3.StringUtils; + +/** GCP native credential shared by the Cloud protobuf and BE Thrift paths. */ +public final class GcpCredentialAdapter implements ObjCredential { + private final GcpCredentialProviderType providerType; + private final String impersonationServiceAccount; + + public GcpCredentialAdapter(GcpCredential credential) { + providerType = credential.getCredentialProviderType(); + impersonationServiceAccount = credential.getImpersonationServiceAccount(); + } + + @Override + public void applyTo(ObjectStoreInfoPB.Builder builder) { + if (!builder.hasProvider()) { + builder.setProvider(ObjectStoreInfoPB.Provider.GCP); + } + GcpCredentialPB.Builder credential = GcpCredentialPB.newBuilder() + .setCredentialProviderType(providerType == GcpCredentialProviderType.DEFAULT + ? GcpCredentialPB.CredentialProviderType.DEFAULT + : GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE); + if (StringUtils.isNotBlank(impersonationServiceAccount)) { + credential.setImpersonationServiceAccount(impersonationServiceAccount); + } + builder.getCredentialBuilder().setGcpCredential(credential); + } + + @Override + public void applyTo(TS3StorageParam param) { + if (!param.isSetProvider()) { + param.setProvider(TObjStorageType.GCP); + } + TGcpCredential credential = new TGcpCredential() + .setCredentialProviderType(providerType == GcpCredentialProviderType.DEFAULT + ? TGcpCredentialProviderType.DEFAULT + : TGcpCredentialProviderType.COMPUTE_ENGINE); + if (StringUtils.isNotBlank(impersonationServiceAccount)) { + credential.setImpersonationServiceAccount(impersonationServiceAccount); + } + param.setCredential(new TCredential().setGcpCredential(credential)); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java new file mode 100644 index 00000000000000..2f6c9794f92eb9 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java @@ -0,0 +1,34 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.property.storage.auth; + +import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.thrift.TS3StorageParam; + +/** + * Provider-native credential that can be serialized for Cloud or BE. + * + *

This transport abstraction is deliberately independent of {@code S3ObjStorage}. + * Runtime S3 client creation is extended through + * {@code AbstractS3CompatibleProperties.createS3Client} instead. + */ +public interface ObjCredential { + void applyTo(ObjectStoreInfoPB.Builder builder); + + void applyTo(TS3StorageParam param); +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java new file mode 100644 index 00000000000000..1e8dc0928f876e --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.property.storage.auth; + +import org.apache.doris.filesystem.auth.GcsAuth; +import org.apache.doris.filesystem.auth.GcsAuthResolver; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** Selects exactly one provider-native credential from object-storage properties. */ +public final class ObjCredentialFactory { + private interface Parser { + Optional parse(Map properties, Optional gcsAuth); + } + + // Add one parser here when introducing OSS, Azure, or another native credential. + // Provider-specific resolvers select credentials; credential classes handle transport conversion. + private static final List PARSERS = Arrays.asList((properties, gcsAuth) -> + gcsAuth.flatMap(GcsAuth::getNativeCredential).map(GcpCredentialAdapter::new)); + + // TODO: Add AWS role-based authentication after its existing property and + // transport fields can be migrated without breaking compatibility. Static + // AK/SK credentials remain outside this provider-native credential model. + + private ObjCredentialFactory() { + } + + public static Optional fromProperties(Map properties) { + return fromProperties(properties, GcsAuthResolver.resolve(properties)); + } + + /** Reuse the authentication choice already resolved by a protocol builder. */ + public static Optional fromProperties(Map properties, Optional gcsAuth) { + ObjCredential selected = null; + for (Parser parser : PARSERS) { + Optional parsed = parser.parse(properties, gcsAuth); + if (!parsed.isPresent()) { + continue; + } + if (selected != null) { + throw new IllegalArgumentException( + "Only one provider-native object storage credential may be configured."); + } + selected = parsed.get(); + } + return Optional.ofNullable(selected); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java index b74680c79d7159..2e179d0bdf32fd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java @@ -21,12 +21,16 @@ import org.apache.doris.cloud.proto.Cloud.CredProviderTypePB; import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB.Provider; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; +import org.apache.doris.datasource.property.storage.auth.ObjCredentialFactory; +import org.apache.doris.filesystem.auth.GcsAuth; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import com.google.common.base.Preconditions; import com.google.common.base.Strings; import java.util.Map; +import java.util.Optional; /** * fe-core-only cloud meta-service glue: builds {@link Cloud.ObjectStoreInfoPB} from the @@ -60,6 +64,10 @@ private CloudObjectStoreAdapter() { /** Direct move of legacy {@code S3Properties.getObjStoreInfoPB}. */ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map properties) { + Optional gcsAuth = GcsAuthResolver.resolve(properties); + if (gcsAuth.filter(GcsAuth::isAnonymous).isPresent()) { + throw new IllegalArgumentException("Anonymous GCS authentication is not supported for storage vaults."); + } Cloud.ObjectStoreInfoPB.Builder builder = Cloud.ObjectStoreInfoPB.newBuilder(); if (properties.containsKey(ENDPOINT)) { builder.setEndpoint(properties.get(ENDPOINT)); @@ -115,6 +123,8 @@ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map credential.applyTo(builder)); return builder; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java index aa313ca01660c4..9dfc255b2b0066 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java @@ -136,6 +136,7 @@ public static void optionalS3Property(Map properties) { } public static void convertToStdProperties(Map properties) { + org.apache.doris.filesystem.auth.GcsAuthResolver.resolve(properties); if (properties.containsKey(Env.ENDPOINT)) { properties.putIfAbsent(ENDPOINT, properties.get(Env.ENDPOINT)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java index 5b979ef1d1fc04..37ff1baed4cd67 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java @@ -18,13 +18,19 @@ package org.apache.doris.datasource.storage; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; +import org.apache.doris.datasource.property.storage.auth.ObjCredentialFactory; +import org.apache.doris.filesystem.auth.GcsAuth; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.thrift.TCredProviderType; +import org.apache.doris.thrift.TObjStorageType; import org.apache.doris.thrift.TS3StorageParam; import org.apache.commons.lang3.StringUtils; +import java.util.Locale; import java.util.Map; +import java.util.Optional; /** * fe-core-only Thrift glue: builds {@link TS3StorageParam} from the persisted user-facing @@ -66,7 +72,13 @@ private S3ThriftAdapter() { /** Direct move of legacy {@code S3Properties.getS3TStorageParam}. */ public static TS3StorageParam getS3TStorageParam(Map properties) { + Optional gcsAuth = GcsAuthResolver.resolve(properties); TS3StorageParam s3Info = new TS3StorageParam(); + if (StringUtils.isNotBlank(properties.get("provider"))) { + String provider = properties.get("provider").trim().toUpperCase(Locale.ROOT); + s3Info.setProvider("S3".equals(provider) ? TObjStorageType.AWS + : "GCS".equals(provider) ? TObjStorageType.GCP : TObjStorageType.valueOf(provider)); + } if (properties.containsKey(ROLE_ARN)) { s3Info.setRoleArn(properties.get(ROLE_ARN)); @@ -95,6 +107,13 @@ public static TS3StorageParam getS3TStorageParam(Map properties) ? DEFAULT_CONNECTION_TIMEOUT_MS : connTimeoutMs)); String usePathStyle = properties.getOrDefault(USE_PATH_STYLE, "false"); s3Info.setUsePathStyle(Boolean.parseBoolean(usePathStyle)); + ObjCredentialFactory.fromProperties(properties, gcsAuth) + .ifPresent(credential -> credential.applyTo(s3Info)); + if (gcsAuth.filter(GcsAuth::isAnonymous).isPresent() + || (gcsAuth.isEmpty() && getCredentialsProviderMode(properties, + AwsCredentialsProviderMode.INSTANCE_PROFILE) == AwsCredentialsProviderMode.ANONYMOUS)) { + s3Info.setCredProviderType(TCredProviderType.ANONYMOUS); + } return s3Info; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index e5df6e21196828..5176038f9843b6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -504,7 +504,8 @@ private Map alignS3FamilyBackendMap(S3CompatibleFileSystemProper // AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS exactly when both AK and SK are blank. aligned.remove("AWS_ROLE_ARN"); aligned.remove("AWS_EXTERNAL_ID"); - if (s3.hasStaticCredentials()) { + if (s3.hasStaticCredentials() + || aligned.containsKey(org.apache.doris.filesystem.auth.GcpCredential.CREDENTIAL_PROVIDER_TYPE)) { aligned.remove("AWS_CREDENTIALS_PROVIDER_TYPE"); } else { aligned.put("AWS_CREDENTIALS_PROVIDER_TYPE", AwsCredentialsProviderMode.ANONYMOUS.name()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java index 19663686f636d4..38fc69b4e23c38 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java @@ -277,7 +277,11 @@ private ExportJob generateExportJob(ConnectContext ctx, Map file exportJob.setWhereExpression(this.expr); exportJob.setWhereStr(this.expr.isPresent() ? this.expr.get().toSql() : ""); // set path - exportJob.setExportPath(this.path); + try { + exportJob.setExportPath(this.brokerDesc.get().getFileLocationForExport(this.path)); + } catch (RuntimeException e) { + throw new AnalysisException("Invalid export path: " + this.path + ", " + e.getMessage(), e); + } // set column separator String columnSeparator = Separator.convertSeparator(fileProperties.getOrDefault( diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java index b029e9695099a5..24326f9ae00fd9 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java @@ -18,9 +18,11 @@ package org.apache.doris.analysis; import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.load.EtlJobType; import org.apache.doris.load.loadv2.BrokerLoadJob; import org.apache.doris.persist.gson.GsonUtils; +import org.apache.doris.thrift.TFileType; import com.google.common.collect.Maps; import org.junit.jupiter.api.Assertions; @@ -75,6 +77,25 @@ public void testBrokerLoadJobRestoreS3StoragePropertiesAfterGsonRoundTrip() thro restoredBrokerDesc.getStorageAdapter().getOrigProps().get("s3.bucket")); } + @Test + public void testGcpNativeCredentialForExportBroker() throws Exception { + Map properties = Maps.newHashMap(); + properties.put("provider", "GCP"); + properties.put("s3.region", "us-east1"); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + + // EXPORT forwards this BrokerDesc map to BE Writer. Verify the shared FE conversion + // preserves the provider-specific credential while keeping the S3-compatible file type. + BrokerDesc brokerDesc = new BrokerDesc("S3", StorageBackend.StorageType.S3, properties); + Assertions.assertEquals("GCS", brokerDesc.getStorageAdapter().getSpiProperties().providerName()); + Assertions.assertEquals(TFileType.FILE_S3, brokerDesc.getFileType()); + Assertions.assertEquals("GCP", brokerDesc.getBackendConfigProperties().get("provider")); + Assertions.assertEquals("DEFAULT", brokerDesc.getBackendConfigProperties() + .get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals("s3://export-bucket/path", + brokerDesc.getFileLocation("gs://export-bucket/path")); + } + private static void setField(Class clazz, Object target, String fieldName, Object value) throws ReflectiveOperationException { Field field = clazz.getDeclaredField(fieldName); diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 57dff9d4d38ed9..e9390b9371ea48 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -23,12 +23,17 @@ import org.apache.doris.common.UserException; import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; import org.apache.doris.datasource.storage.S3ResourceCompat; +import org.apache.doris.datasource.storage.S3ThriftAdapter; +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.meta.MetaContext; import org.apache.doris.mysql.privilege.AccessControllerManager; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.commands.CreateResourceCommand; import org.apache.doris.nereids.trees.plans.commands.info.CreateResourceInfo; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.thrift.TS3StorageParam; import com.google.common.base.Strings; import com.google.common.collect.ImmutableMap; @@ -43,11 +48,16 @@ import java.io.DataInputStream; import java.io.DataOutputStream; +import java.lang.reflect.Field; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.util.HashMap; import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; public class S3ResourceTest { private static final Logger LOG = LogManager.getLogger(S3ResourceTest.class); @@ -213,6 +223,69 @@ public void testSerialization() throws Exception { Files.deleteIfExists(path); } + @Test + public void testConcurrentAltersPreserveBothUpdates() throws Exception { + CountDownLatch firstReady = new CountDownLatch(1); + CountDownLatch releaseFirst = new CountDownLatch(1); + CountDownLatch secondStarted = new CountDownLatch(1); + AtomicBoolean first = new AtomicBoolean(true); + AtomicReference failure = new AtomicReference<>(); + S3Resource resource = new S3Resource("concurrent_alter") { + @Override + public void writeLock() { + if (first.compareAndSet(true, false)) { + firstReady.countDown(); + try { + Assertions.assertTrue(releaseFirst.await(10, TimeUnit.SECONDS)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } + super.writeLock(); + } + }; + resource.setProperties(ImmutableMap.copyOf(s3Properties)); + Thread updateConnections = new Thread(() -> { + try { + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.MAX_CONNECTIONS, "77")); + } catch (Throwable e) { + failure.compareAndSet(null, e); + } + }); + Thread updateTimeout = new Thread(() -> { + secondStarted.countDown(); + try { + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.REQUEST_TIMEOUT_MS, "8888")); + } catch (Throwable e) { + failure.compareAndSet(null, e); + } + }); + try { + updateConnections.start(); + Assertions.assertTrue(firstReady.await(10, TimeUnit.SECONDS)); + updateTimeout.start(); + Assertions.assertTrue(secondStarted.await(10, TimeUnit.SECONDS)); + // The second ALTER either waits for the first one's monitor, or (before the fix) + // finishes and is subsequently overwritten by the first one's stale snapshot. + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (updateTimeout.isAlive() && updateTimeout.getState() != Thread.State.BLOCKED + && System.nanoTime() < deadline) { + Thread.yield(); + } + Assertions.assertTrue(!updateTimeout.isAlive() || updateTimeout.getState() == Thread.State.BLOCKED); + } finally { + releaseFirst.countDown(); + updateConnections.join(10000); + updateTimeout.join(10000); + } + Assertions.assertFalse(updateConnections.isAlive()); + Assertions.assertFalse(updateTimeout.isAlive()); + Assertions.assertNull(failure.get()); + Assertions.assertEquals("77", resource.getProperty(S3ResourceCompat.MAX_CONNECTIONS)); + Assertions.assertEquals("8888", resource.getProperty(S3ResourceCompat.REQUEST_TIMEOUT_MS)); + } + @Test public void testModifyProperties() throws Exception { ImmutableMap properties = ImmutableMap.of( @@ -323,4 +396,372 @@ public void testPingS3WithRoleArn() { Assertions.assertTrue(false, e.getMessage()); } } + + @Test + public void testGsEndpointAndAlterAliases() throws Exception { + Map properties = new HashMap<>(); + properties.put("provider", "gcp"); + properties.put("s3.endpoint", "https://old.example.com"); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put("s3.region", "us-central1"); + properties.put("s3.bucket", "bucket"); + properties.put("s3.root.path", "prefix"); + properties.put("s3.access_key", "access"); + properties.put("s3.secret_key", "secret"); + properties.put("s3_validity_check", "false"); + properties.put("type", "s3"); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Assertions.assertEquals("us-central1", resource.getProperty(S3ResourceCompat.REGION)); + Assertions.assertEquals("bucket", resource.getProperty(S3ResourceCompat.BUCKET)); + Assertions.assertEquals("prefix", resource.getProperty(S3ResourceCompat.ROOT_PATH)); + Assertions.assertEquals("https://storage.googleapis.com", resource.getProperty(S3ResourceCompat.ENDPOINT)); + Assertions.assertEquals("https://storage.googleapis.com", + S3ThriftAdapter.getS3TStorageParam(resource.getCopiedProperties()).getEndpoint()); + Assertions.assertFalse(resource.getCopiedProperties().containsKey("gs.endpoint")); + resource.modifyProperties(ImmutableMap.of("gs.endpoint", "https://custom.example.com", + "s3_validity_check", "false")); + Assertions.assertEquals("https://custom.example.com", resource.getProperty(S3ResourceCompat.ENDPOINT)); + resource.modifyProperties(ImmutableMap.of("s3.endpoint", "https://storage.googleapis.com", + "s3_validity_check", "false")); + Assertions.assertEquals("https://storage.googleapis.com", resource.getProperty(S3ResourceCompat.ENDPOINT)); + Assertions.assertEquals("https://storage.googleapis.com", properties.get("gs.endpoint")); + Assertions.assertEquals("https://old.example.com", properties.get("s3.endpoint")); + } + + @Test + public void testGsEndpointForStorageVault(@Mocked Env env) throws Exception { + for (String provider : new String[] {"GCP", "gcp", "S3", "OSS"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", provider); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put("gs.access_key", "gcp-access"); + properties.put("gs.secret_key", "gcp-secret"); + properties.put("gs.use_path_style", "true"); + properties.put("s3.root.path", "prefix"); + CreateResourceCommand command = new CreateResourceCommand( + new CreateResourceInfo(false, false, "gcp_vault", ImmutableMap.copyOf(properties))); + command.getInfo().analyzeResourceType(); + S3StorageVault vault = new S3StorageVault("gcp_vault", false, false, command); + vault.checkCreationProperties(ImmutableMap.copyOf(properties)); + Assertions.assertEquals("prefix", vault.getCopiedProperties().get(S3ResourceCompat.ROOT_PATH)); + String expectedEndpoint = "GCP".equalsIgnoreCase(provider) + ? "https://storage.googleapis.com" : s3Endpoint; + Assertions.assertEquals(expectedEndpoint, + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getEndpoint()); + Assertions.assertEquals("GCP".equalsIgnoreCase(provider) ? "gcp-access" : s3AccessKey, + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getAk()); + Assertions.assertEquals("GCP".equalsIgnoreCase(provider) ? "gcp-secret" : s3SecretKey, + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getSk()); + Assertions.assertEquals("GCP".equalsIgnoreCase(provider), + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getUsePathStyle()); + } + } + + @Test + public void testStorageVaultRejectsNonAllowlistedAliases() throws Exception { + StorageVaultMgr manager = new StorageVaultMgr(null); + for (String key : new String[] {"gs.endpoint", S3ResourceCompat.ENDPOINT, S3ResourceCompat.Env.ENDPOINT, + "gs.access_key", "gs.secret_key", "gs.session_token", "gs.connection.maximum", + "gs.connection.request.timeout", "gs.connection.timeout", "gs.use_path_style", + "gs.force_parsing_by_standard_uri"}) { + try { + manager.alterStorageVault(StorageVault.StorageVaultType.S3, + ImmutableMap.of(key, "https://custom.example.com"), "gcp_vault"); + Assertions.fail("Storage Vault must reject properties outside its ALTER whitelist: " + key); + } catch (IllegalArgumentException exception) { + Assertions.assertEquals("Alter property " + key + " is not allowed.", exception.getMessage()); + } + } + } + + @Test + public void testBlankGsEndpointPreservesResourceAliases() throws Exception { + for (String blank : new String[] {"", " \t"}) { + for (String endpointKey : new String[] {S3ResourceCompat.ENDPOINT, S3ResourceCompat.Env.ENDPOINT}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + properties.remove(S3ResourceCompat.Env.ENDPOINT); + properties.put(endpointKey, s3Endpoint); + properties.put("gs.endpoint", blank); + properties.put("gs.connection.timeout", blank); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.ENDPOINT)); + Assertions.assertFalse(resource.getCopiedProperties().containsKey("gs.endpoint")); + Assertions.assertEquals(Integer.parseInt(s3ConnTimeoutMs), + S3ThriftAdapter.getS3TStorageParam(resource.getCopiedProperties()).getConnTimeoutMs()); + Map updated = new HashMap<>(); + updated.put("gs.endpoint", blank); + updated.put("s3_validity_check", "false"); + resource.modifyProperties(updated); + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.ENDPOINT)); + Assertions.assertEquals(blank, updated.get("gs.endpoint")); + Assertions.assertFalse(resource.getCopiedProperties().containsKey("gs.connection.timeout")); + } + } + } + + @Test + public void testGsEndpointCannotAlterPolicyResource() throws Exception { + S3Resource resource = new S3Resource("gcp_resource"); + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.references.put("policy", Resource.ReferenceType.POLICY); + try { + resource.modifyProperties(ImmutableMap.of("gs.endpoint", "https://custom.example.com")); + Assertions.fail("A gs.endpoint alias must not bypass the policy endpoint restriction"); + } catch (DdlException exception) { + Assertions.assertTrue(exception.getMessage().contains(S3ResourceCompat.ENDPOINT)); + } + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.ENDPOINT)); + } + + @Test + public void testGsEndpointDoesNotOverrideOtherProviders() throws Exception { + for (String provider : new String[] {"S3", "OSS", "AZURE", ""}) { + Map properties = new HashMap<>(s3Properties); + if (!provider.isEmpty()) { + properties.put("provider", provider); + } + properties.put("gs.endpoint", "https://storage.googleapis.com"); + S3Resource resource = new S3Resource("other_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.ENDPOINT)); + resource.modifyProperties(ImmutableMap.of("gs.endpoint", "https://custom.example.com")); + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.ENDPOINT)); + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.Env.ENDPOINT)); + } + } + + @Test + public void testGsEndpointUsesEffectiveProviderOnAlter() throws Exception { + S3Resource resource = new S3Resource("gcp_resource"); + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.modifyProperties(ImmutableMap.of("provider", "S3", "gs.endpoint", "https://ignored.example.com")); + Assertions.assertEquals("S3", resource.getProperty("provider")); + Assertions.assertEquals(s3Endpoint, resource.getProperty(S3ResourceCompat.ENDPOINT)); + resource.modifyProperties(ImmutableMap.of("provider", "gcp", "gs.endpoint", "https://storage.googleapis.com")); + Assertions.assertEquals("gcp", resource.getProperty("provider")); + Assertions.assertEquals("https://storage.googleapis.com", resource.getProperty(S3ResourceCompat.ENDPOINT)); + resource.modifyProperties(ImmutableMap.of("provider", "", "gs.endpoint", "https://custom.example.com")); + Assertions.assertEquals("gcp", resource.getProperty("provider")); + Assertions.assertEquals("https://custom.example.com", resource.getProperty(S3ResourceCompat.ENDPOINT)); + } + + @Test + public void testGsAliasesMatchConnectorAndResourceProtocols() throws Exception { + for (String provider : new String[] {"GCP", "gcp"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", provider); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put("gs.access_key", "gcp-access"); + properties.put("gs.secret_key", "gcp-secret"); + properties.put("gs.session_token", "gcp-token"); + properties.put("gs.connection.maximum", "123"); + properties.put("gs.connection.request.timeout", "456"); + properties.put("gs.connection.timeout", "789"); + properties.put("gs.use_path_style", "true"); + properties.put("gs.force_parsing_by_standard_uri", "true"); + properties.put(S3ResourceCompat.SECRET_KEY, "other-secret"); + properties.put(S3ResourceCompat.CONNECTION_TIMEOUT_MS, "999"); + Map original = new HashMap<>(properties); + S3CompatibleFileSystemProperties connector = (S3CompatibleFileSystemProperties) StorageAdapter.of(properties).getSpiProperties(); + + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Map normalized = resource.getCopiedProperties(); + Assertions.assertFalse(normalized.keySet().stream().anyMatch(key -> key.startsWith("gs."))); + Assertions.assertEquals("true", normalized.get("force_parsing_by_standard_uri")); + TS3StorageParam thrift = S3ThriftAdapter.getS3TStorageParam(normalized); + Assertions.assertEquals(connector.getEndpoint(), thrift.getEndpoint()); + Assertions.assertEquals(connector.getAccessKey(), thrift.getAk()); + Assertions.assertEquals(connector.getSecretKey(), thrift.getSk()); + Assertions.assertEquals(connector.getSessionToken(), thrift.getToken()); + Assertions.assertEquals(Integer.parseInt(connector.getMaxConnections()), thrift.getMaxConn()); + Assertions.assertEquals(Integer.parseInt(connector.getRequestTimeoutMs()), thrift.getRequestTimeoutMs()); + Assertions.assertEquals(Integer.parseInt(connector.getConnectionTimeoutMs()), thrift.getConnTimeoutMs()); + Assertions.assertEquals(Boolean.parseBoolean(connector.getUsePathStyle()), thrift.isUsePathStyle()); + Assertions.assertEquals(connector.getAccessKey(), CloudObjectStoreAdapter.getObjStoreInfoPB(normalized).getAk()); + Assertions.assertEquals(connector.getSecretKey(), CloudObjectStoreAdapter.getObjStoreInfoPB(normalized).getSk()); + Assertions.assertEquals(thrift.isUsePathStyle(), CloudObjectStoreAdapter.getObjStoreInfoPB(normalized).getUsePathStyle()); + Assertions.assertEquals(original, properties); + + resource.modifyProperties(ImmutableMap.of("gs.secret_key", "new-secret", + "gs.connection.timeout", "321", "gs.use_path_style", "false")); + thrift = S3ThriftAdapter.getS3TStorageParam(resource.getCopiedProperties()); + Assertions.assertEquals("new-secret", thrift.getSk()); + Assertions.assertEquals(321, thrift.getConnTimeoutMs()); + Assertions.assertFalse(thrift.isUsePathStyle()); + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.SECRET_KEY, "canonical-secret", + S3ResourceCompat.SESSION_TOKEN, "")); + connector = (S3CompatibleFileSystemProperties) StorageAdapter.of(resource.getCopiedProperties()).getSpiProperties(); + Assertions.assertEquals("canonical-secret", connector.getSecretKey()); + Assertions.assertEquals("", connector.getSessionToken()); + } + } + + @Test + public void testBlankGsCredentialAndConnectionAliases() throws Exception { + for (String blank : new String[] {"", " \t"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + properties.put("gs.access_key", blank); + properties.put("gs.secret_key", blank); + properties.put("gs.session_token", blank); + properties.put(S3ResourceCompat.SESSION_TOKEN, "canonical-token"); + properties.put("gs.connection.maximum", blank); + properties.put(S3ResourceCompat.MAX_CONNECTIONS, "123"); + properties.put("gs.use_path_style", blank); + properties.put(S3ResourceCompat.USE_PATH_STYLE, "true"); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.modifyProperties(ImmutableMap.of("gs.secret_key", blank, "gs.connection.maximum", blank)); + TS3StorageParam thrift = S3ThriftAdapter.getS3TStorageParam(resource.getCopiedProperties()); + Assertions.assertEquals(s3AccessKey, thrift.getAk()); + Assertions.assertEquals(s3SecretKey, thrift.getSk()); + Assertions.assertEquals("canonical-token", thrift.getToken()); + Assertions.assertEquals(123, thrift.getMaxConn()); + Assertions.assertTrue(thrift.isUsePathStyle()); + Assertions.assertFalse(resource.getCopiedProperties().keySet().stream().anyMatch(key -> key.startsWith("gs."))); + } + } + + @Test + public void testGsAliasesDoNotOverrideOtherProviders() throws Exception { + for (String provider : new String[] {"S3", "OSS", "AZURE", ""}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", provider); + properties.put("gs.access_key", "ignored-access"); + properties.put("gs.secret_key", "ignored-secret"); + properties.put("gs.connection.timeout", "789"); + properties.put("gs.use_path_style", "true"); + S3Resource resource = new S3Resource("other_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.modifyProperties(ImmutableMap.of("gs.secret_key", "also-ignored", + "gs.connection.timeout", "321")); + TS3StorageParam thrift = S3ThriftAdapter.getS3TStorageParam(resource.getCopiedProperties()); + Assertions.assertEquals(s3AccessKey, thrift.getAk()); + Assertions.assertEquals(s3SecretKey, thrift.getSk()); + Assertions.assertEquals(Integer.parseInt(s3ConnTimeoutMs), thrift.getConnTimeoutMs()); + Assertions.assertFalse(thrift.isUsePathStyle()); + } + } + + @Test + public void testGsCredentialAliasesCannotBypassNativeAuthValidation() throws Exception { + Map properties = new HashMap<>(); + properties.put("provider", "GCP"); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put(S3ResourceCompat.BUCKET, "bucket"); + properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + for (String key : new String[] {"gs.access_key", "gs.secret_key", "gs.session_token", + S3ResourceCompat.ACCESS_KEY, S3ResourceCompat.SECRET_KEY, S3ResourceCompat.SESSION_TOKEN}) { + S3Resource resource = new S3Resource("gcp_resource"); + Map conflicting = new HashMap<>(properties); + conflicting.put(key, "conflicting-credential"); + Assertions.assertThrows(IllegalArgumentException.class, + () -> resource.setProperties(ImmutableMap.copyOf(conflicting))); + + resource.setProperties(ImmutableMap.copyOf(properties)); + Map before = resource.getCopiedProperties(); + Assertions.assertThrows(IllegalArgumentException.class, + () -> resource.modifyProperties(ImmutableMap.of(key, "conflicting-credential"))); + Assertions.assertEquals(before, resource.getCopiedProperties()); + Assertions.assertEquals("DEFAULT", resource.getProperty(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + } + S3Resource resource = new S3Resource("gcp_resource"); + properties.remove(GcpCredential.CREDENTIAL_PROVIDER_TYPE); + properties.put("gs.access_key", "access"); + properties.put("gs.secret_key", "secret"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Assertions.assertThrows(IllegalArgumentException.class, () -> resource.modifyProperties( + ImmutableMap.of(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"))); + // Empty AK/SK updates are ignored by Resource ALTER and cannot clear stored credentials. + Assertions.assertThrows(IllegalArgumentException.class, () -> resource.modifyProperties( + ImmutableMap.of(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT", + S3ResourceCompat.ACCESS_KEY, "", S3ResourceCompat.SECRET_KEY, ""))); + } + + @Test + public void testGsAliasesTakePrecedenceAfterMergingAlter() throws Exception { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + // Simulate a resource persisted before gs.* normalization was introduced. + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + Map legacy = resource.getCopiedProperties(); + legacy.put("gs.secret_key", "old-secret"); + legacy.put("gs.session_token", "old-token"); + legacy.put("gs.connection.timeout", "789"); + field.set(resource, legacy); + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.SECRET_KEY, "new-secret", + S3ResourceCompat.SESSION_TOKEN, "")); + Map normalized = resource.getCopiedProperties(); + S3CompatibleFileSystemProperties connector = (S3CompatibleFileSystemProperties) StorageAdapter.of(normalized).getSpiProperties(); + Assertions.assertEquals("old-secret", connector.getSecretKey()); + Assertions.assertEquals("old-token", connector.getSessionToken()); + Assertions.assertEquals(connector.getSecretKey(), S3ThriftAdapter.getS3TStorageParam(normalized).getSk()); + Assertions.assertEquals(connector.getSessionToken(), S3ThriftAdapter.getS3TStorageParam(normalized).getToken()); + Assertions.assertEquals(connector.getSecretKey(), CloudObjectStoreAdapter.getObjStoreInfoPB(normalized).getSk()); + Assertions.assertEquals(789, S3ThriftAdapter.getS3TStorageParam(normalized).getConnTimeoutMs()); + Assertions.assertFalse(normalized.containsKey("gs.secret_key")); + Assertions.assertFalse(normalized.containsKey("gs.session_token")); + Assertions.assertFalse(normalized.containsKey("gs.connection.timeout")); + + // Within the same ALTER input, gs.* also takes precedence over s3.*. + resource.modifyProperties(ImmutableMap.of("gs.secret_key", "updated-gs-secret", + S3ResourceCompat.SECRET_KEY, "ignored-s3-secret")); + Assertions.assertEquals("updated-gs-secret", resource.getProperty(S3ResourceCompat.SECRET_KEY)); + } + + @Test + public void testBlankStoredGsAliasesFallBackToAlterValues() throws Exception { + for (String blank : new String[] {"", " \t"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + Map legacy = resource.getCopiedProperties(); + legacy.put("gs.secret_key", blank); + legacy.put("gs.session_token", blank); + field.set(resource, legacy); + + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.SECRET_KEY, "new-secret", + S3ResourceCompat.SESSION_TOKEN, "")); + Map normalized = resource.getCopiedProperties(); + S3CompatibleFileSystemProperties connector = (S3CompatibleFileSystemProperties) StorageAdapter.of(normalized).getSpiProperties(); + Assertions.assertEquals("new-secret", connector.getSecretKey()); + Assertions.assertEquals("", connector.getSessionToken()); + Assertions.assertEquals(connector.getSecretKey(), S3ThriftAdapter.getS3TStorageParam(normalized).getSk()); + Assertions.assertFalse(normalized.containsKey("gs.secret_key")); + Assertions.assertFalse(normalized.containsKey("gs.session_token")); + } + } + + @Test + public void testStoredGsEndpointCannotChangePolicyResourceOnAlter() throws Exception { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.references.put("policy", Resource.ReferenceType.POLICY); + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + Map legacy = resource.getCopiedProperties(); + legacy.put("gs.endpoint", "https://different.example.com"); + field.set(resource, legacy); + + Assertions.assertThrows(DdlException.class, () -> resource.modifyProperties( + ImmutableMap.of(S3ResourceCompat.SECRET_KEY, "new-secret"))); + Assertions.assertEquals(legacy, resource.getCopiedProperties()); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java new file mode 100644 index 00000000000000..9b8d168de4d14b --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java @@ -0,0 +1,253 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.cloud.catalog; + +import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.StorageVault; +import org.apache.doris.catalog.StorageVault.StorageVaultType; +import org.apache.doris.catalog.StorageVaultMgr; +import org.apache.doris.cloud.proto.Cloud; +import org.apache.doris.cloud.rpc.MetaServiceProxy; +import org.apache.doris.common.Config; +import org.apache.doris.common.DdlException; +import org.apache.doris.datasource.storage.S3ResourceCompat; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; +import org.apache.doris.system.SystemInfoService; + +import mockit.Mock; +import mockit.MockUp; +import mockit.Mocked; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.atomic.AtomicReference; + +public class S3StorageVaultTest { + private static final String ROLE = "arn:aws:iam::123456789012:role/test-role"; + private static final String ACCOUNT = "target@my-project.iam.gserviceaccount.com"; + + @Mocked + private Env env; + + // Capture the actual CREATE/ALTER RPC, then stop before cache updates and backend synchronization. + private Cloud.ObjectStoreInfoPB captureRequest(Map auth, boolean isAlter) throws Exception { + AtomicReference captured = new AtomicReference<>(); + new MockUp() { + @Mock + public Cloud.AlterObjStoreInfoResponse alterStorageVault(Cloud.AlterObjStoreInfoRequest request) { + captured.set(request); + return Cloud.AlterObjStoreInfoResponse.newBuilder().setStatus( + Cloud.MetaServiceResponseStatus.newBuilder().setCode(Cloud.MetaServiceCode.INVALID_ARGUMENT) + .setMsg("captured vault request")).build(); + } + }; + String oldEndpoint = Config.meta_service_endpoint; + String oldCloudId = Config.cloud_unique_id; + Config.meta_service_endpoint = "127.0.0.1:20121"; + Config.cloud_unique_id = "vault-default-test"; + try { + StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); + if (isAlter) { + Assertions.assertThrows(DdlException.class, + () -> mgr.alterStorageVault(StorageVaultType.S3, new HashMap<>(auth), "iam_vault")); + } else { + Map properties = new HashMap<>(auth); + boolean isGcp = "GCP".equals(auth.get("provider")) + || auth.keySet().stream().anyMatch(key -> key.toLowerCase().startsWith("gs.")); + properties.put("type", "S3"); + properties.put("provider", isGcp ? "GCP" : "S3"); + properties.put(S3ResourceCompat.ENDPOINT, isGcp ? "storage.googleapis.com" : "s3.us-east-1.amazonaws.com"); + properties.put(S3ResourceCompat.REGION, "us-east-1"); + properties.put(S3ResourceCompat.BUCKET, "test-bucket"); + properties.put(S3ResourceCompat.ROOT_PATH, "test-root"); + properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); + CreateStorageVaultCommand command = new CreateStorageVaultCommand(false, "iam_vault", properties); + command.setStorageVaultType(StorageVaultType.S3); + StorageVault vault = StorageVault.fromCommand(command); + Assertions.assertThrows(DdlException.class, () -> mgr.createS3Vault(vault)); + } + Assertions.assertNotNull(captured.get(), "The request must reach the MS RPC boundary"); + Assertions.assertEquals(isAlter ? Cloud.AlterObjStoreInfoRequest.Operation.ALTER_S3_VAULT + : Cloud.AlterObjStoreInfoRequest.Operation.ADD_S3_VAULT, captured.get().getOp()); + if (isAlter) { + Assertions.assertFalse(captured.get().getVault().getObjInfo().hasProvider(), + "ALTER must not send an inferred immutable provider"); + } + return captured.get().getVault().getObjInfo(); + } finally { + Config.meta_service_endpoint = oldEndpoint; + Config.cloud_unique_id = oldCloudId; + } + } + + @Test + public void testAwsCreatePreservesDefaultAndExplicitProvider() throws Exception { + Map properties = new HashMap<>(); + properties.put(S3ResourceCompat.ROLE_ARN, ROLE); + Cloud.ObjectStoreInfoPB obj = captureRequest(properties, false); + Assertions.assertTrue(obj.hasCredProviderType()); + Assertions.assertEquals(Cloud.CredProviderTypePB.INSTANCE_PROFILE, obj.getCredProviderType()); + for (String key : new String[] {S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, + S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE}) { + Map explicit = new HashMap<>(properties); + explicit.put(key, "CONTAINER"); + obj = captureRequest(explicit, false); + Assertions.assertTrue(obj.hasCredProviderType()); + Assertions.assertEquals(Cloud.CredProviderTypePB.CONTAINER, obj.getCredProviderType()); + } + } + + @Test + public void testAwsAlterPreservesOmissionAndExplicitClears() throws Exception { + Map properties = new HashMap<>(); + properties.put(S3ResourceCompat.ROLE_ARN, ROLE); + Assertions.assertFalse(captureRequest(properties, true).hasCredProviderType()); + properties.put(S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, "DEFAULT"); + Cloud.ObjectStoreInfoPB obj = captureRequest(properties, true); + Assertions.assertTrue(obj.hasCredProviderType()); + Assertions.assertEquals(Cloud.CredProviderTypePB.DEFAULT, obj.getCredProviderType()); + properties.clear(); + properties.put(S3ResourceCompat.ROLE_ARN, ""); + properties.put(S3ResourceCompat.EXTERNAL_ID, ""); + obj = captureRequest(properties, true); + Assertions.assertTrue(obj.hasRoleArn()); + Assertions.assertEquals("", obj.getRoleArn()); + Assertions.assertTrue(obj.hasExternalId()); + Assertions.assertEquals("", obj.getExternalId()); + Assertions.assertFalse(obj.hasCredProviderType()); + } + + @Test + public void testGcpCreateDefaultsAndExplicitProvider() throws Exception { + Map properties = new HashMap<>(); + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ACCOUNT); + Cloud.GcpCredentialPB credential = captureRequest(properties, false).getCredential().getGcpCredential(); + Assertions.assertTrue(credential.hasCredentialProviderType()); + Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.DEFAULT, + credential.getCredentialProviderType()); + Assertions.assertEquals(ACCOUNT, credential.getImpersonationServiceAccount()); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "COMPUTE_ENGINE"); + credential = captureRequest(properties, false).getCredential().getGcpCredential(); + Assertions.assertTrue(credential.hasCredentialProviderType()); + Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE, + credential.getCredentialProviderType()); + } + + @Test + public void testGcpEmptyTargetDefaultsToDefault() throws Exception { + Map properties = new HashMap<>(); + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ""); + Cloud.GcpCredentialPB credential = captureRequest(properties, false).getCredential().getGcpCredential(); + Assertions.assertTrue(credential.hasCredentialProviderType()); + Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.DEFAULT, + credential.getCredentialProviderType()); + Assertions.assertFalse(credential.hasImpersonationServiceAccount()); + + credential = captureRequest(properties, true).getCredential().getGcpCredential(); + Assertions.assertFalse(credential.hasCredentialProviderType()); + Assertions.assertTrue(credential.hasImpersonationServiceAccount()); + Assertions.assertEquals("", credential.getImpersonationServiceAccount()); + } + + @Test + public void testGcpAlterPreservesOmittedFields() throws Exception { + Map properties = new HashMap<>(); + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ACCOUNT); + Cloud.GcpCredentialPB credential = captureRequest(properties, true).getCredential().getGcpCredential(); + Assertions.assertFalse(credential.hasCredentialProviderType()); + Assertions.assertTrue(credential.hasImpersonationServiceAccount()); + Assertions.assertEquals(ACCOUNT, credential.getImpersonationServiceAccount()); + properties.clear(); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "COMPUTE_ENGINE"); + credential = captureRequest(properties, true).getCredential().getGcpCredential(); + Assertions.assertTrue(credential.hasCredentialProviderType()); + Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE, + credential.getCredentialProviderType()); + Assertions.assertFalse(credential.hasImpersonationServiceAccount()); + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ACCOUNT); + credential = captureRequest(properties, true).getCredential().getGcpCredential(); + Assertions.assertTrue(credential.hasCredentialProviderType()); + Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE, + credential.getCredentialProviderType()); + Assertions.assertTrue(credential.hasImpersonationServiceAccount()); + Assertions.assertEquals(ACCOUNT, credential.getImpersonationServiceAccount()); + properties.clear(); + properties.put(S3ResourceCompat.USE_PATH_STYLE, "true"); + Cloud.ObjectStoreInfoPB obj = captureRequest(properties, true); + Assertions.assertFalse(obj.hasCredential()); + Assertions.assertFalse(obj.hasCredProviderType()); + } + + @Test + public void testGcpCreateWithoutAuthenticationDefaultsToAdc() throws Exception { + Map properties = new HashMap<>(); + properties.put("provider", "GCP"); + Cloud.ObjectStoreInfoPB obj = captureRequest(properties, false); + Assertions.assertTrue(obj.hasCredential()); + Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.DEFAULT, + obj.getCredential().getGcpCredential().getCredentialProviderType()); + Assertions.assertFalse(obj.getCredential().getGcpCredential().hasImpersonationServiceAccount()); + } + + @Test + public void testGcpAnonymousAliasesAreRejectedBeforeCreateRpc() throws Exception { + new MockUp() { + @Mock + public Cloud.AlterObjStoreInfoResponse alterStorageVault(Cloud.AlterObjStoreInfoRequest request) { + throw new AssertionError("Anonymous GCS vaults must be rejected before the MS RPC"); + } + }; + StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); + for (String key : new String[] {GcpCredential.CREDENTIAL_PROVIDER_TYPE, + S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE}) { + Map properties = new HashMap<>(); + properties.put("type", "S3"); + properties.put("provider", "GCP"); + properties.put(S3ResourceCompat.ENDPOINT, "storage.googleapis.com"); + properties.put(S3ResourceCompat.REGION, "us-east1"); + properties.put(S3ResourceCompat.BUCKET, "test-bucket"); + properties.put(S3ResourceCompat.ROOT_PATH, "test-root"); + properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); + properties.put(key, " anonymous "); + CreateStorageVaultCommand command = new CreateStorageVaultCommand(false, "anonymous_gcp", properties); + command.setStorageVaultType(StorageVaultType.S3); + StorageVault vault = StorageVault.fromCommand(command); + IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, + () -> mgr.createS3Vault(vault), key); + Assertions.assertTrue(error.getMessage().contains("storage vaults"), key); + } + } + + @Test + public void testGcpStorageOnlyAlterDoesNotInjectAdc() throws Exception { + Map properties = new HashMap<>(); + properties.put("s3.endpoint", "storage.googleapis.com"); + // Inspect the builder directly: ALTER's public API may reject immutable storage fields. + java.lang.reflect.Method method = StorageVaultMgr.class.getDeclaredMethod( + "buildAlterS3VaultRequest", Map.class, String.class); + method.setAccessible(true); + StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); + Cloud.StorageVaultPB.Builder request = (Cloud.StorageVaultPB.Builder) method.invoke(mgr, properties, "gcp"); + Assertions.assertFalse(request.getObjInfo().hasCredential()); + Assertions.assertFalse(request.getObjInfo().hasProvider()); + } + +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/storage/ObjectInfoAdapterTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/storage/ObjectInfoAdapterTest.java index ed26ad4a0aab41..af1ac11b37e4ea 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/cloud/storage/ObjectInfoAdapterTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/storage/ObjectInfoAdapterTest.java @@ -83,4 +83,20 @@ public void testOssStageBindingCarriesEveryField() { Assertions.assertEquals("stage-sk", oss.getSecretKey()); Assertions.assertEquals("stage-token", oss.getSessionToken()); } + + @Test + public void testNativeGcpStagePreservesCredentialSourceAndImpersonation() { + Cloud.ObjectStoreInfoPB.Builder info = Cloud.ObjectStoreInfoPB.newBuilder() + .setProvider(Cloud.ObjectStoreInfoPB.Provider.GCP) + .setEndpoint("storage.googleapis.com").setRegion("us-east1").setBucket("stage-bucket"); + info.getCredentialBuilder().getGcpCredentialBuilder() + .setCredentialProviderType(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE) + .setImpersonationServiceAccount("target@test.iam.gserviceaccount.com"); + StorageAdapter adapter = ObjectInfoAdapter.toStorageAdapter(new ObjectInfo(info.build())); + Assertions.assertEquals("COMPUTE_ENGINE", + adapter.getBackendConfigProperties().get("gs.credential_provider_type")); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", + adapter.getBackendConfigProperties().get("gs.impersonation_service_account")); + Assertions.assertFalse(adapter.getBackendConfigProperties().containsKey("AWS_CREDENTIALS_PROVIDER_TYPE")); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java index 0d1c80af22c565..ef9a6925772f82 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java @@ -17,6 +17,7 @@ package org.apache.doris.nereids.trees.plans; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator; @@ -35,8 +36,11 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; import java.lang.reflect.Field; +import java.util.Map; public class OutFileTest extends TestWithFeService implements PlanPatternMatchSupported { private final NereidsParser parser = new NereidsParser(); @@ -94,9 +98,109 @@ public void testHdfsOutFileCarriesDefaultFs() throws Exception { Field field = ResultFileSink.class.getDeclaredField("fileSinkOptions"); field.setAccessible(true); TResultFileSinkOptions sinkOptions = (TResultFileSinkOptions) field.get(fragment.getSink()); + Assertions.assertEquals("hdfs://127.0.0.1:8020/tmp/outfile_test_", sinkOptions.getFilePath()); Assertions.assertEquals("hdfs://127.0.0.1:8020", sinkOptions.getBrokerProperties().get("fs.defaultFS")); } + @Test + public void testGcsOutFileNormalizesPathAndPreservesCredentials() throws Exception { + for (String scheme : new String[] {"gs", "s3"}) { + for (String provider : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + for (String serviceAccount : new String[] {"", "target@test-project.iam.gserviceaccount.com"}) { + String properties = "\"provider\" = \"GCP\"," + + "\"s3.endpoint\" = \"https://storage.googleapis.com\"," + + "\"s3.region\" = \"us-central1\"," + + "\"" + GcpCredential.CREDENTIAL_PROVIDER_TYPE + "\" = \"" + provider + "\""; + if (!serviceAccount.isEmpty()) { + properties += ",\"" + GcpCredential.IMPERSONATION_SERVICE_ACCOUNT + + "\" = \"" + serviceAccount + "\""; + } + TResultFileSinkOptions options = getFileSinkOptions("select * from T1 into outfile '" + + scheme + "://outfile-bucket/nested/result_' format as csv properties (" + + properties + ")"); + Assertions.assertEquals("s3://outfile-bucket/nested/result_", options.getFilePath()); + Map backendProperties = options.getBrokerProperties(); + Assertions.assertEquals("GCP", backendProperties.get("provider")); + Assertions.assertEquals("https://storage.googleapis.com", backendProperties.get("AWS_ENDPOINT")); + // GCSProperties does not bind s3.region and retains its compatibility default. + Assertions.assertEquals("us-east1", backendProperties.get("AWS_REGION")); + Assertions.assertEquals(provider, backendProperties.get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals(serviceAccount, + backendProperties.getOrDefault(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "")); + } + } + } + } + + @ParameterizedTest + @CsvSource({ + "http, false", "https, false", "http, true", "https, true", + "HTTP, false", "HTTPS, false" + }) + public void testS3HttpOutFilePreservesPath(String scheme, boolean usePathStyle) throws Exception { + String path = scheme + "://s3.us-east-1.amazonaws.com/outfile-bucket/nested/result_"; + TResultFileSinkOptions options = getFileSinkOptions("select * from T1 into outfile '" + + path + "' format as csv properties (" + + "\"s3.endpoint\" = \"https://s3.us-east-1.amazonaws.com\"," + + "\"s3.region\" = \"us-east-1\"," + + "\"s3.access_key\" = \"test-ak\",\"s3.secret_key\" = \"test-sk\"," + + "\"use_path_style\" = \"" + usePathStyle + "\")"); + Assertions.assertEquals(path, options.getFilePath()); + } + + @Test + public void testHdfsOutFilePreservesPathWithoutAuthority() throws Exception { + String path = "hdfs:/tmp/outfile_test_"; + TResultFileSinkOptions options = getFileSinkOptions("select * from T1 into outfile '" + + path + "' format as csv properties (\"hadoop.username\" = \"doris\")"); + Assertions.assertEquals(path, options.getFilePath()); + } + + @Test + public void testS3OutFilePreservesPath() throws Exception { + TResultFileSinkOptions options = getFileSinkOptions("select * from T1 into outfile '" + + "s3://outfile-bucket/nested/result_' format as csv properties (" + + "\"s3.endpoint\" = \"https://s3.us-east-1.amazonaws.com\"," + + "\"s3.region\" = \"us-east-1\"," + + "\"s3.access_key\" = \"test-ak\",\"s3.secret_key\" = \"test-sk\")"); + Assertions.assertEquals("s3://outfile-bucket/nested/result_", options.getFilePath()); + Assertions.assertEquals("test-ak", options.getBrokerProperties().get("AWS_ACCESS_KEY")); + Assertions.assertEquals("test-sk", options.getBrokerProperties().get("AWS_SECRET_KEY")); + } + + @ParameterizedTest + @CsvSource({ + "oss://outfile-bucket, https://oss-cn-hangzhou.aliyuncs.com, cn-hangzhou", + "oss://outfile-bucket.oss-cn-hangzhou.aliyuncs.com, https://oss-cn-hangzhou.aliyuncs.com, cn-hangzhou", + "s3://outfile-bucket.oss-cn-hangzhou.aliyuncs.com, https://oss-cn-hangzhou.aliyuncs.com, cn-hangzhou", + "cos://outfile-bucket, https://cos.ap-guangzhou.myqcloud.com, ap-guangzhou", + "cosn://outfile-bucket, https://cos.ap-guangzhou.myqcloud.com, ap-guangzhou", + "obs://outfile-bucket, https://obs.cn-north-4.myhuaweicloud.com, cn-north-4", + "bos://outfile-bucket, https://s3.us-east-1.amazonaws.com, us-east-1", + "s3a://outfile-bucket, https://s3.us-east-1.amazonaws.com, us-east-1", + "s3n://outfile-bucket, https://s3.us-east-1.amazonaws.com, us-east-1" + }) + public void testS3CompatibleOutFileNormalizesPath(String location, String endpoint, String region) throws Exception { + TResultFileSinkOptions options = getFileSinkOptions("select * from T1 into outfile '" + + location + "/nested/result_' format as csv properties (" + + "\"s3.endpoint\" = \"" + endpoint + "\"," + + "\"s3.region\" = \"" + region + "\"," + + "\"s3.access_key\" = \"test-ak\",\"s3.secret_key\" = \"test-sk\")"); + Assertions.assertEquals("s3://outfile-bucket/nested/result_", options.getFilePath()); + Assertions.assertEquals(endpoint, options.getBrokerProperties().get("AWS_ENDPOINT")); + Assertions.assertEquals(region, options.getBrokerProperties().get("AWS_REGION")); + Assertions.assertEquals("test-ak", options.getBrokerProperties().get("AWS_ACCESS_KEY")); + Assertions.assertEquals("test-sk", options.getBrokerProperties().get("AWS_SECRET_KEY")); + } + + private TResultFileSinkOptions getFileSinkOptions(String sql) throws Exception { + PlanFragment fragment = getOutputFragment(sql); + Assertions.assertTrue(fragment.getSink() instanceof ResultFileSink); + Field field = ResultFileSink.class.getDeclaredField("fileSinkOptions"); + field.setAccessible(true); + return (TResultFileSinkOptions) field.get(fragment.getSink()); + } + private PlanFragment getOutputFragment(String sql) throws Exception { StatementScopeIdGenerator.clear(); StatementContext statementContext = MemoTestUtils.createStatementContext(connectContext, sql); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java index 981b7be805b898..aefbe2900c6fff 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java @@ -17,18 +17,132 @@ package org.apache.doris.nereids.trees.plans.commands; +import org.apache.doris.analysis.BrokerDesc; +import org.apache.doris.analysis.StorageBackend.StorageType; +import org.apache.doris.analysis.TableName; import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.load.ExportJob; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.utframe.TestWithFeService; +import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import java.lang.reflect.Method; import java.util.Collections; +import java.util.HashMap; import java.util.Map; import java.util.Optional; -public class ExportCommandTest { +public class ExportCommandTest extends TestWithFeService { + @Override + protected void runBeforeAll() throws Exception { + createDatabase("export_test"); + connectContext.setDatabase("export_test"); + createTable("CREATE TABLE T1 (id bigint) DUPLICATE KEY(id) " + + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES (\"replication_num\" = \"1\")"); + } + + @ParameterizedTest + @CsvSource({ + "s3://export-bucket, https://s3.us-east-1.amazonaws.com, us-east-1", + "oss://export-bucket, https://oss-cn-hangzhou.aliyuncs.com, cn-hangzhou", + "oss://export-bucket.oss-cn-hangzhou.aliyuncs.com, https://oss-cn-hangzhou.aliyuncs.com, cn-hangzhou", + "s3://export-bucket.oss-cn-hangzhou.aliyuncs.com, https://oss-cn-hangzhou.aliyuncs.com, cn-hangzhou", + "cos://export-bucket, https://cos.ap-guangzhou.myqcloud.com, ap-guangzhou", + "cosn://export-bucket, https://cos.ap-guangzhou.myqcloud.com, ap-guangzhou", + "obs://export-bucket, https://obs.cn-north-4.myhuaweicloud.com, cn-north-4", + "bos://export-bucket, https://s3.us-east-1.amazonaws.com, us-east-1", + "s3a://export-bucket, https://s3.us-east-1.amazonaws.com, us-east-1", + "s3n://export-bucket, https://s3.us-east-1.amazonaws.com, us-east-1" + }) + public void testS3CompatibleExportNormalizesPath(String location, String endpoint, String region) throws Exception { + BrokerDesc broker = new BrokerDesc(null, s3Properties(endpoint, region)); + ExportJob job = generateExportJob(location + "/nested/result_", broker); + Assertions.assertEquals("s3://export-bucket/nested/result_", job.getExportPath()); + Map backendProperties = job.getBrokerDesc().getBackendConfigProperties(); + Assertions.assertEquals(endpoint, backendProperties.get("AWS_ENDPOINT")); + Assertions.assertEquals(region, backendProperties.get("AWS_REGION")); + Assertions.assertEquals("test-ak", backendProperties.get("AWS_ACCESS_KEY")); + Assertions.assertEquals("test-sk", backendProperties.get("AWS_SECRET_KEY")); + } + + @Test + public void testGcsExportNormalizesPathAndPreservesCredentials() throws Exception { + for (String scheme : new String[] {"gs", "s3"}) { + for (String provider : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + for (String serviceAccount : new String[] {"", "target@test-project.iam.gserviceaccount.com"}) { + Map properties = new HashMap<>(); + properties.put("provider", "GCP"); + properties.put("s3.endpoint", "https://storage.googleapis.com"); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, provider); + if (!serviceAccount.isEmpty()) { + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, serviceAccount); + } + ExportJob job = generateExportJob(scheme + "://export-bucket/nested/result_", + new BrokerDesc(null, properties)); + Assertions.assertEquals("s3://export-bucket/nested/result_", job.getExportPath()); + Map backendProperties = job.getBrokerDesc().getBackendConfigProperties(); + Assertions.assertEquals("GCP", backendProperties.get("provider")); + Assertions.assertEquals("https://storage.googleapis.com", backendProperties.get("AWS_ENDPOINT")); + Assertions.assertEquals(provider, backendProperties.get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals(serviceAccount, + backendProperties.getOrDefault(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "")); + } + } + } + } + + @ParameterizedTest + @CsvSource({ + "http, false", "https, false", "http, true", "https, true", + "HTTP, false", "HTTPS, false" + }) + public void testHttpExportPreservesPath(String scheme, boolean usePathStyle) throws Exception { + Map properties = s3Properties("https://s3.us-east-1.amazonaws.com", "us-east-1"); + properties.put("use_path_style", Boolean.toString(usePathStyle)); + String path = scheme + "://s3.us-east-1.amazonaws.com/export-bucket/nested/result_"; + Assertions.assertEquals(path, generateExportJob(path, new BrokerDesc(null, properties)).getExportPath()); + } + + @Test + public void testNonS3ExportPreservesPath() throws Exception { + String hdfsPath = "hdfs:/tmp/export_test_"; + Assertions.assertEquals(hdfsPath, generateExportJob(hdfsPath, + new BrokerDesc(null, ImmutableMap.of("hadoop.username", "doris"))).getExportPath()); + String brokerPath = "oss://export-bucket/nested/result_"; + Assertions.assertEquals(brokerPath, generateExportJob(brokerPath, + new BrokerDesc("broker", StorageType.BROKER, ImmutableMap.of())).getExportPath()); + String localPath = "file:///tmp/export_test_"; + Assertions.assertEquals(localPath, generateExportJob(localPath, + new BrokerDesc("local", StorageType.LOCAL, null)).getExportPath()); + } + + private Map s3Properties(String endpoint, String region) { + Map properties = new HashMap<>(); + properties.put("s3.endpoint", endpoint); + properties.put("s3.region", region); + properties.put("s3.access_key", "test-ak"); + properties.put("s3.secret_key", "test-sk"); + return properties; + } + + private ExportJob generateExportJob(String path, BrokerDesc broker) throws Exception { + ExportCommand command = new ExportCommand(ImmutableList.of("T1"), ImmutableList.of(), Optional.empty(), + path, ImmutableMap.of(), Optional.of(broker)); + // Inspect the job before registration so the test never starts an export task. + Method method = ExportCommand.class.getDeclaredMethod("generateExportJob", + ConnectContext.class, Map.class, TableName.class); + method.setAccessible(true); + return (ExportJob) method.invoke(command, connectContext, ImmutableMap.of(), + new TableName("internal", "export_test", "T1")); + } @Test void testEnableInt96TimestampsProperty() { diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java index 60ee030dcf3624..c101eb6f8aae08 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java @@ -22,6 +22,7 @@ import org.apache.doris.common.Pair; import org.apache.doris.datasource.property.fileformat.CsvFileFormatProperties; import org.apache.doris.datasource.property.fileformat.DeferredFileFormatProperties; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.analyzer.UnboundSlot; import org.apache.doris.nereids.load.NereidsDataDescription; @@ -84,6 +85,46 @@ protected void runBeforeAll() throws Exception { createTable(createTableBitmapSql); } + @Test + public void testGcpNativeCredentialForS3Load() { + String loadSql = "LOAD LABEL gcp_native_load(" + + " DATA INFILE(\"gs://load-bucket/customer\")" + + " INTO TABLE customer" + + ") WITH S3(" + + " \"provider\" = \"GCP\"," + + " \"gs.endpoint\" = \"https://storage.googleapis.com\"," + + " \"gs.credential_provider_type\" = \"COMPUTE_ENGINE\"," + + " \"gs.impersonation_service_account\" = " + + "\"target@my-project.iam.gserviceaccount.com\"" + + ");"; + + LoadCommand command = (LoadCommand) new NereidsParser().parseSingle(loadSql); + BrokerDesc brokerDesc = command.getBrokerDesc(); + Assertions.assertEquals("us-east1", brokerDesc.getBackendConfigProperties().get("AWS_REGION")); + Assertions.assertEquals("GCP", brokerDesc.getBackendConfigProperties().get("provider")); + Assertions.assertEquals("COMPUTE_ENGINE", brokerDesc.getBackendConfigProperties() + .get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals("target@my-project.iam.gserviceaccount.com", + brokerDesc.getBackendConfigProperties().get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + } + + @Test + public void testGcpNativeCredentialValidationHappensBeforeLoadSubmission() { + String loadSql = "LOAD LABEL invalid_gcp_native_load(" + + " DATA INFILE(\"gs://load-bucket/customer\")" + + " INTO TABLE customer" + + ") WITH S3(" + + " \"provider\" = \"GCP\"," + + " \"gs.credential_provider_type\" = \"\"" + + ");"; + + IllegalArgumentException exception = Assertions.assertThrows(IllegalArgumentException.class, + () -> new NereidsParser().parseSingle(loadSql)); + Assertions.assertTrue(exception.getMessage().contains(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertTrue(exception.getMessage().contains("DEFAULT")); + Assertions.assertTrue(exception.getMessage().contains("COMPUTE_ENGINE")); + } + @Test public void testLoadCommandBitmap() { String loadSql1 = "LOAD LABEL load_bitmap_table_test( " diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java index 6c12f41e3793ba..a1486eafac1e58 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java @@ -24,9 +24,11 @@ import org.apache.doris.catalog.Type; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Config; +import org.apache.doris.common.FeConstants; import org.apache.doris.common.Pair; import org.apache.doris.common.util.FileFormatConstants; import org.apache.doris.common.util.FileFormatUtils; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.proto.Types.PScalarType; import org.apache.doris.proto.Types.PStructField; import org.apache.doris.proto.Types.PTypeNode; @@ -105,6 +107,54 @@ public void testHiveParquetTimeZoneRejectsAmbiguousShortAlias() { Assertions.assertTrue(exception.getMessage().contains("short timezone aliases are not supported")); } + @Test + public void testGcpNativeCredentialForS3Tvf() throws AnalysisException { + boolean previousRunningUnitTest = FeConstants.runningUnitTest; + FeConstants.runningUnitTest = true; + try { + Map properties = Maps.newHashMap(); + properties.put("uri", "gs://tvf-bucket/path/file.parquet"); + properties.put("format", "parquet"); + properties.put("provider", "GCP"); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "COMPUTE_ENGINE"); + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, + "target@my-project.iam.gserviceaccount.com"); + + S3TableValuedFunction tvf = new S3TableValuedFunction(properties); + Assertions.assertEquals("s3://tvf-bucket/path/file.parquet", tvf.getFilePath()); + Assertions.assertEquals("us-east1", tvf.getBackendConnectProperties().get("AWS_REGION")); + Assertions.assertEquals("GCP", tvf.getBackendConnectProperties().get("provider")); + Assertions.assertEquals("COMPUTE_ENGINE", tvf.getBackendConnectProperties() + .get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals("target@my-project.iam.gserviceaccount.com", + tvf.getBackendConnectProperties().get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + } finally { + FeConstants.runningUnitTest = previousRunningUnitTest; + } + } + + @Test + public void testGcpNativeCredentialValidationHappensBeforeTvfAccess() { + boolean previousRunningUnitTest = FeConstants.runningUnitTest; + FeConstants.runningUnitTest = true; + try { + Map properties = Maps.newHashMap(); + properties.put("uri", "gs://tvf-bucket/path/file.parquet"); + properties.put("format", "parquet"); + properties.put("provider", "GCP"); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + properties.put("s3.session_token", "session-token"); + + IllegalArgumentException exception = Assertions.assertThrows(IllegalArgumentException.class, + () -> new S3TableValuedFunction(properties)); + Assertions.assertTrue(exception.getMessage().contains("session token")); + } finally { + FeConstants.runningUnitTest = previousRunningUnitTest; + } + + } + @Test public void testCsvSchemaParse() { Config.enable_date_conversion = true; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java new file mode 100644 index 00000000000000..873ccd5f5b2faa --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + + + +/** GCP native credential shared by the Cloud protobuf and BE Thrift paths. */ +public final class GcpCredential { + public static final String CREDENTIAL_PROVIDER_TYPE = "gs.credential_provider_type"; + public static final String IMPERSONATION_SERVICE_ACCOUNT = "gs.impersonation_service_account"; + + private static final String SERVICE_ACCOUNT_PATTERN = + "^[^@\\s]+@(?:[^@\\s]+\\.iam\\.gserviceaccount\\.com|" + + "developer\\.gserviceaccount\\.com|appspot\\.gserviceaccount\\.com)$"; + + private final GcpCredentialProviderType providerType; + private final String impersonationServiceAccount; + + public GcpCredential(GcpCredentialProviderType providerType, String impersonationServiceAccount) { + this.providerType = providerType; + this.impersonationServiceAccount = impersonationServiceAccount == null ? "" : impersonationServiceAccount; + validate(); + } + + public GcpCredentialProviderType getCredentialProviderType() { + return providerType; + } + + public String getImpersonationServiceAccount() { + return impersonationServiceAccount; + } + + private void validate() { + if (providerType == GcpCredentialProviderType.ANONYMOUS) { + throw new IllegalArgumentException("Anonymous access does not use a native GCP credential"); + } + if (impersonationServiceAccount.isEmpty()) { + return; + } + if (!impersonationServiceAccount.matches(SERVICE_ACCOUNT_PATTERN)) { + throw new IllegalArgumentException("Invalid GCP service account email: " + + impersonationServiceAccount); + } + } + +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java new file mode 100644 index 00000000000000..21fe7a85fa6a0b --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java @@ -0,0 +1,35 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import java.util.Locale; + +public enum GcpCredentialProviderType { + DEFAULT, + COMPUTE_ENGINE, + ANONYMOUS; + + public static GcpCredentialProviderType fromString(String propertyName, String value) { + try { + return valueOf(value.trim().toUpperCase(Locale.ROOT)); + } catch (RuntimeException e) { + throw new IllegalArgumentException("Invalid " + propertyName + " value: " + value + + ". Only DEFAULT, COMPUTE_ENGINE and ANONYMOUS are supported.", e); + } + } +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java new file mode 100644 index 00000000000000..ee233eb5b0d74a --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java @@ -0,0 +1,50 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import java.util.Optional; + +/** Immutable authentication choice shared by GCS clients, cache identity and protocol builders. */ +public final class GcsAuth { + public enum Mode { + HMAC, ADC, COMPUTE_ENGINE, ANONYMOUS + } + + private final Mode mode; + private final GcpCredential nativeCredential; + + GcsAuth(Mode mode, String impersonationServiceAccount) { + this.mode = mode; + this.nativeCredential = mode == Mode.ADC || mode == Mode.COMPUTE_ENGINE + ? new GcpCredential(mode == Mode.ADC ? GcpCredentialProviderType.DEFAULT + : GcpCredentialProviderType.COMPUTE_ENGINE, impersonationServiceAccount) + : null; + } + + public Mode getMode() { + return mode; + } + + public boolean isAnonymous() { + return mode == Mode.ANONYMOUS; + } + + public Optional getNativeCredential() { + return Optional.ofNullable(nativeCredential); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java new file mode 100644 index 00000000000000..ba963ceb00b35f --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java @@ -0,0 +1,169 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + + +import java.util.Locale; +import java.util.Map; +import java.util.Optional; + +/** + * Resolves GCS authentication without creating clients or loading credentials. + * Vault ALTER still preserves field presence when constructing its patch from this result. + */ +public final class GcsAuthResolver { + private static final String CREDENTIAL_PROVIDER_TYPE = GcpCredential.CREDENTIAL_PROVIDER_TYPE; + private static final String IMPERSONATION_SERVICE_ACCOUNT = GcpCredential.IMPERSONATION_SERVICE_ACCOUNT; + private static final String[] ACCESS_KEYS = {"gs.access_key", "s3.access_key", "AWS_ACCESS_KEY", "access_key"}; + private static final String[] SECRET_KEYS = {"gs.secret_key", "s3.secret_key", "AWS_SECRET_KEY", "secret_key"}; + private static final String[] TOKENS = {"gs.session_token", "s3.session_token", "AWS_TOKEN", "session_token"}; + private static final String[] AWS_PROVIDERS = {"s3.credentials_provider_type", "AWS_CREDENTIALS_PROVIDER_TYPE"}; + private static final String[] AWS_ROLES = {"s3.role_arn", "AWS_ROLE_ARN", "s3.external_id", "AWS_EXTERNAL_ID"}; + + private GcsAuthResolver() { + } + + public static Optional resolve(Map properties) { + boolean hasNativeProperties = hasNativeCredentialProperties(properties); + boolean hasGcsSelector = guessIsGcs(properties) + || "true".equalsIgnoreCase(getPropertyIgnoreCase(properties, "fs.gcs.support")); + boolean hasAccessKey = hasNonBlankProperty(properties, ACCESS_KEYS); + boolean hasSecretKey = hasNonBlankProperty(properties, SECRET_KEYS); + boolean hasLegacyAnonymous = hasLegacyAnonymousProvider(properties); + String provider = getPropertyIgnoreCase(properties, "provider"); + boolean useLegacyAnonymousDefault = "S3".equalsIgnoreCase(provider) + && !hasNonBlankProperty(properties, AWS_PROVIDERS); + if (isNotBlank(provider) && !"GCP".equalsIgnoreCase(provider) && !"GCS".equalsIgnoreCase(provider)) { + if (hasNativeProperties) { + throw new IllegalArgumentException("Native GCP authentication requires provider=GCP, but found: " + + provider); + } + // Legacy GCS configurations may explicitly select the S3-compatible protocol. + // Preserve their HMAC/anonymous authentication without enabling native ADC for S3. + if (!"S3".equalsIgnoreCase(provider) || !hasGcsSelector + || !(hasAccessKey || hasSecretKey || hasLegacyAnonymous || useLegacyAnonymousDefault)) { + return Optional.empty(); + } + } + if (!hasNativeProperties && !hasGcsSelector) { + return Optional.empty(); + } + + String providerType = getPropertyIgnoreCase(properties, CREDENTIAL_PROVIDER_TYPE); + String impersonation = getPropertyIgnoreCase(properties, IMPERSONATION_SERVICE_ACCOUNT); + GcsAuth.Mode mode; + if (hasNativeProperties) { + GcpCredentialProviderType type = providerType == null ? GcpCredentialProviderType.DEFAULT + : GcpCredentialProviderType.fromString(CREDENTIAL_PROVIDER_TYPE, providerType); + mode = type == GcpCredentialProviderType.ANONYMOUS ? GcsAuth.Mode.ANONYMOUS + : type == GcpCredentialProviderType.COMPUTE_ENGINE ? GcsAuth.Mode.COMPUTE_ENGINE : GcsAuth.Mode.ADC; + } else if (hasLegacyAnonymous) { + mode = GcsAuth.Mode.ANONYMOUS; + } else if (hasAccessKey || hasSecretKey) { + mode = GcsAuth.Mode.HMAC; + } else if (useLegacyAnonymousDefault) { + mode = GcsAuth.Mode.ANONYMOUS; + } else { + mode = GcsAuth.Mode.ADC; + } + + // The storage properties validate paired HMAC keys. Raw Vault patches may update only one key. + if (mode != GcsAuth.Mode.HMAC) { + if (hasAccessKey || hasSecretKey || hasNonBlankProperty(properties, TOKENS)) { + throw new IllegalArgumentException(CREDENTIAL_PROVIDER_TYPE + + " cannot be used together with access key, secret key, or session token."); + } + if (hasNonBlankProperty(properties, AWS_ROLES) + || ((hasNativeProperties || mode != GcsAuth.Mode.ANONYMOUS) + && hasNonBlankProperty(properties, AWS_PROVIDERS))) { + throw new IllegalArgumentException("Native GCP authentication cannot be used together with " + + "AWS role, external ID, or credentials provider properties."); + } + } + if (mode == GcsAuth.Mode.ANONYMOUS && impersonation != null && !impersonation.isEmpty()) { + throw new IllegalArgumentException("ANONYMOUS cannot be used with " + IMPERSONATION_SERVICE_ACCOUNT); + } + return Optional.of(new GcsAuth(mode, impersonation)); + } + + public static boolean hasNativeCredentialProperties(Map properties) { + return properties.keySet().stream().anyMatch(key -> CREDENTIAL_PROVIDER_TYPE.equalsIgnoreCase(key) + || IMPERSONATION_SERVICE_ACCOUNT.equalsIgnoreCase(key)); + } + + // Storage selection also uses these signals, before authentication is resolved. + public static boolean guessIsGcs(Map properties) { + for (Map.Entry entry : properties.entrySet()) { + String key = entry.getKey().toLowerCase(Locale.ROOT); + String value = entry.getValue(); + if ("provider".equals(key) && ("GCP".equalsIgnoreCase(value) || "GCS".equalsIgnoreCase(value))) { + return true; + } + if ((CREDENTIAL_PROVIDER_TYPE.equals(key) || IMPERSONATION_SERVICE_ACCOUNT.equals(key)) + && isNotBlank(value)) { + return true; + } + if (("uri".equals(key) || "warehouse".equals(key)) + && value != null && value.regionMatches(true, 0, "gs://", 0, 5)) { + return true; + } + if ("gs.endpoint".equals(key) && isNotBlank(value)) { + return true; + } + if (("s3.endpoint".equals(key) || "aws_endpoint".equals(key) || "endpoint".equals(key)) + && value != null && value.toLowerCase(Locale.ROOT).endsWith("storage.googleapis.com")) { + return true; + } + } + return false; + } + + private static boolean isNotBlank(String value) { + return value != null && !value.isBlank(); + } + + private static String trim(String value) { + return value == null ? null : value.trim(); + } + + private static boolean hasLegacyAnonymousProvider(Map properties) { + for (String name : AWS_PROVIDERS) { + if ("ANONYMOUS".equalsIgnoreCase(trim(getPropertyIgnoreCase(properties, name)))) { + return true; + } + } + return false; + } + + private static boolean hasNonBlankProperty(Map properties, String... names) { + for (String name : names) { + if (isNotBlank(getPropertyIgnoreCase(properties, name))) { + return true; + } + } + return false; + } + + private static String getPropertyIgnoreCase(Map properties, String name) { + return properties.entrySet().stream() + .filter(entry -> name.equalsIgnoreCase(entry.getKey())) + .map(Map.Entry::getValue) + .findFirst() + .orElse(null); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java new file mode 100644 index 00000000000000..b9ca61e6cc9b77 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java @@ -0,0 +1,126 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +public class GcsAuthResolverTest { + @Test + public void testDefaultAndExplicitModes() { + Map props = new HashMap<>(); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + props.put("provider", "GCP"); + Assertions.assertEquals(GcsAuth.Mode.ADC, GcsAuthResolver.resolve(props).get().getMode()); + for (String provider : new String[] {"DEFAULT", "COMPUTE_ENGINE", "ANONYMOUS"}) { + props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, provider); + GcsAuth auth = GcsAuthResolver.resolve(props).get(); + Assertions.assertEquals(provider.equals("DEFAULT") ? GcsAuth.Mode.ADC : GcsAuth.Mode.valueOf(provider), + auth.getMode()); + Assertions.assertEquals(!provider.equals("ANONYMOUS"), auth.getNativeCredential().isPresent()); + } + } + + @Test + public void testImpersonationAndFactoryUseResolvedCredentials() { + Map props = new HashMap<>(); + props.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "target@test.iam.gserviceaccount.com"); + GcsAuth auth = GcsAuthResolver.resolve(props).get(); + Assertions.assertEquals(GcsAuth.Mode.ADC, auth.getMode()); + GcpCredential credential = auth.getNativeCredential().orElseThrow(); + Assertions.assertEquals(auth.getNativeCredential().get().getCredentialProviderType(), + credential.getCredentialProviderType()); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", credential.getImpersonationServiceAccount()); + } + + @Test + public void testHmacAliasesDoNotAcquireAdc() { + for (String[] keys : new String[][] {{"gs.access_key", "gs.secret_key"}, + {"s3.access_key", "s3.secret_key"}, {"AWS_ACCESS_KEY", "AWS_SECRET_KEY"}, + {"ACCESS_KEY", "SECRET_KEY"}}) { + Map props = new HashMap<>(); + props.put("provider", "GCP"); + props.put(keys[0], "ak"); + props.put(keys[1], "sk"); + Assertions.assertEquals(GcsAuth.Mode.HMAC, GcsAuthResolver.resolve(props).get().getMode()); + Assertions.assertFalse(GcsAuthResolver.resolve(props).get().getNativeCredential().isPresent()); + // Vault ALTER may supply just one HMAC field; it must not inject native credentials. + props.remove(keys[1]); + Assertions.assertEquals(GcsAuth.Mode.HMAC, GcsAuthResolver.resolve(props).get().getMode()); + } + } + + @Test + public void testLegacyS3HmacRequiresGcsSelector() { + Map props = new HashMap<>(); + props.put("provider", "S3"); + props.put("s3.access_key", "ak"); + props.put("s3.secret_key", "sk"); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + props.put("s3.endpoint", "storage.googleapis.com"); + Assertions.assertEquals(GcsAuth.Mode.HMAC, GcsAuthResolver.resolve(props).get().getMode()); + for (String provider : new String[] {"AZURE", "OSS"}) { + props.put("provider", provider); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + } + props.put("provider", "S3"); + props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + } + + @Test + public void testLegacyS3OmittedAuthenticationUsesAnonymousOnlyForGcs() { + Map props = new HashMap<>(); + props.put("provider", "s3"); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + props.put("s3.endpoint", "https://storage.googleapis.com"); + GcsAuth auth = GcsAuthResolver.resolve(props).get(); + Assertions.assertEquals(GcsAuth.Mode.ANONYMOUS, auth.getMode()); + Assertions.assertFalse(auth.getNativeCredential().isPresent()); + + props.put("s3.credentials_provider_type", "DEFAULT"); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + props.remove("s3.credentials_provider_type"); + props.put("provider", "GCP"); + Assertions.assertEquals(GcsAuth.Mode.ADC, GcsAuthResolver.resolve(props).get().getMode()); + props.remove("provider"); + Assertions.assertEquals(GcsAuth.Mode.ADC, GcsAuthResolver.resolve(props).get().getMode()); + } + + @Test + public void testOtherProvidersAndConflictingNativeProperties() { + for (String provider : new String[] {"S3", "AZURE", "OSS"}) { + Map props = new HashMap<>(); + props.put("provider", provider); + props.put("s3.endpoint", "storage.googleapis.com"); + if ("S3".equals(provider)) { + Assertions.assertEquals(GcsAuth.Mode.ANONYMOUS, GcsAuthResolver.resolve(props).get().getMode()); + } else { + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + } + props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + } + Map props = new HashMap<>(); + props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, ""); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml index 6c276177773547..d798c46d9ca285 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml @@ -36,6 +36,11 @@ under the License. client in fe-filesystem-s3-base. + + com.google.auth + google-auth-library-oauth2-http + 1.46.0 + org.apache.doris fe-filesystem-spi diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java new file mode 100644 index 00000000000000..9c1a38f48d1907 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java @@ -0,0 +1,148 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.gcs; + +import org.apache.doris.filesystem.auth.GcpCredential; + +import com.google.auth.oauth2.ComputeEngineCredentials; +import com.google.auth.oauth2.GoogleCredentials; +import com.google.auth.oauth2.ImpersonatedCredentials; +import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider; +import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration; +import software.amazon.awssdk.core.client.config.SdkAdvancedClientOption; +import software.amazon.awssdk.core.interceptor.Context; +import software.amazon.awssdk.core.interceptor.ExecutionAttributes; +import software.amazon.awssdk.core.interceptor.ExecutionInterceptor; +import software.amazon.awssdk.core.retry.RetryPolicy; +import software.amazon.awssdk.core.retry.backoff.EqualJitterBackoffStrategy; +import software.amazon.awssdk.core.signer.NoOpSigner; +import software.amazon.awssdk.http.SdkHttpRequest; +import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3AsyncClient; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3Configuration; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.net.URI; +import java.time.Duration; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +/** Creates GCP OAuth2 credentials and adapts them to the S3-compatible client. */ +public final class GcpAuth { + public static final String DEVSTORAGE_SCOPE = + "https://www.googleapis.com/auth/devstorage.read_write"; + private static final String CLOUD_PLATFORM_SCOPE = "https://www.googleapis.com/auth/cloud-platform"; + private static final int IMPERSONATION_LIFETIME_SECONDS = 3600; + + private GcpAuth() { + } + + public static S3Client createS3Client(GcsFileSystemProperties properties) throws IOException { + return S3Client.builder() + .httpClient(UrlConnectionHttpClient.builder().socketTimeout(Duration.ofSeconds(30)) + .connectionTimeout(Duration.ofSeconds(30)).build()) + .endpointOverride(endpoint(properties)) + .credentialsProvider(AnonymousCredentialsProvider.create()) + .region(Region.of(properties.getRegion())) + .overrideConfiguration(clientConfiguration(properties)) + .serviceConfiguration(serviceConfiguration(properties)) + .build(); + } + + public static S3AsyncClient createS3AsyncClient(GcsFileSystemProperties properties) throws IOException { + return S3AsyncClient.builder() + .endpointOverride(endpoint(properties)) + .credentialsProvider(AnonymousCredentialsProvider.create()) + .region(Region.of(properties.getRegion())) + .overrideConfiguration(clientConfiguration(properties)) + .serviceConfiguration(serviceConfiguration(properties)) + .build(); + } + + private static URI endpoint(GcsFileSystemProperties properties) { + String endpoint = properties.getEndpoint(); + return URI.create(endpoint.contains("://") ? endpoint : "https://" + endpoint); + } + + private static S3Configuration serviceConfiguration(GcsFileSystemProperties properties) { + return S3Configuration.builder().chunkedEncodingEnabled(false) + .pathStyleAccessEnabled(properties.isUsePathStyle()).build(); + } + + private static ClientOverrideConfiguration clientConfiguration(GcsFileSystemProperties properties) + throws IOException { + return ClientOverrideConfiguration.builder() + .retryPolicy(RetryPolicy.builder().numRetries(3) + .backoffStrategy(EqualJitterBackoffStrategy.builder() + .baseDelay(Duration.ofSeconds(1)) + .maxBackoffTime(Duration.ofMinutes(1)).build()).build()) + .putAdvancedOption(SdkAdvancedClientOption.SIGNER, new NoOpSigner()) + .addExecutionInterceptor(bearerTokenInterceptor(createCredentials( + properties.getAuth().getNativeCredential().orElseThrow()))) + .build(); + } + + public static GoogleCredentials createCredentials(GcpCredential credential) throws IOException { + GoogleCredentials source; + switch (credential.getCredentialProviderType()) { + case DEFAULT: + source = GoogleCredentials.getApplicationDefault(); + break; + case COMPUTE_ENGINE: + source = ComputeEngineCredentials.create(); + break; + default: + throw new IllegalArgumentException("Unsupported GCP credential provider: " + + credential.getCredentialProviderType()); + } + if (credential.getImpersonationServiceAccount().isEmpty()) { + return source.createScoped(Collections.singletonList(DEVSTORAGE_SCOPE)); + } + // The source token calls the IAM Credentials API; the impersonated + // target token itself only needs object read/write access. + source = source.createScoped(Collections.singletonList(CLOUD_PLATFORM_SCOPE)); + return ImpersonatedCredentials.create(source, credential.getImpersonationServiceAccount(), null, + Collections.singletonList(DEVSTORAGE_SCOPE), IMPERSONATION_LIFETIME_SECONDS); + } + + public static ExecutionInterceptor bearerTokenInterceptor(GoogleCredentials credentials) { + return new ExecutionInterceptor() { + @Override + public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, + ExecutionAttributes executionAttributes) { + try { + Map> metadata = credentials.getRequestMetadata(); + List authorization = metadata.get("Authorization"); + if (authorization == null || authorization.isEmpty() + || authorization.get(0) == null || authorization.get(0).isEmpty()) { + throw new IOException("GCP credentials did not provide an Authorization header"); + } + return context.httpRequest().toBuilder() + .putHeader("Authorization", authorization.get(0)) + .build(); + } catch (IOException e) { + throw new UncheckedIOException("Failed to obtain a GCP access token", e); + } + } + }; + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index f24e7b3b77a49c..b2b7e0f9bbf209 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -17,6 +17,11 @@ package org.apache.doris.filesystem.gcs; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.GcpCredentialProviderType; +import org.apache.doris.filesystem.auth.GcsAuth; +import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.filesystem.properties.FsCacheKeys; import org.apache.doris.filesystem.s3.AbstractDelegatingS3Properties; import org.apache.doris.foundation.property.ConnectorProperty; @@ -26,12 +31,11 @@ import java.util.Set; /** - * Provider-owned GCS properties (S3 interoperability / XML API, HMAC credentials only). + * Provider-owned GCS properties supporting HMAC, native OAuth2 and anonymous access. * *

The public aliases, ordering, defaults, and static-credential validation follow fe-core - * GCSProperties. The Google native SDK (JSON API) is intentionally NOT used: it only accepts - * OAuth2 credentials, while Doris GCS configurations carry HMAC keys. AWS-only credential - * mechanisms (role ARN, instance profile) are rejected at binding time. + * GCSProperties. Object access uses the S3-compatible XML API; native IAM Hadoop operations + * use the GCS connector. AWS-only role and instance-profile options are rejected at binding time. */ public final class GcsFileSystemProperties extends AbstractDelegatingS3Properties { @@ -105,12 +109,24 @@ public final class GcsFileSystemProperties extends AbstractDelegatingS3Propertie description = "Whether to force standard URI parsing.") private String forceParsingByStandardUrl = "false"; + private GcsAuth auth; + + @ConnectorProperty(names = {GcpCredential.CREDENTIAL_PROVIDER_TYPE}, required = false, + description = "GCP credential source: DEFAULT, COMPUTE_ENGINE or ANONYMOUS; defaults to ADC without HMAC.") + private String credentialProviderType = ""; + + @ConnectorProperty(names = {GcpCredential.IMPERSONATION_SERVICE_ACCOUNT}, required = false, + description = "Target service account email used for GCP impersonation.") + private String impersonationServiceAccount = ""; + private GcsFileSystemProperties(Map rawProperties) { super(rawProperties); } public static GcsFileSystemProperties of(Map properties) { GcsFileSystemProperties props = new GcsFileSystemProperties(properties); + props.auth = GcsAuthResolver.resolve(properties) + .orElseGet(() -> GcsAuthResolver.resolve(withGcpProvider(properties)).orElseThrow()); props.bindAndCollect(); props.validate(); return props; @@ -134,10 +150,58 @@ public Map toMap() { return Collections.unmodifiableMap(kv); } + private static Map withGcpProvider(Map properties) { + Map selected = new HashMap<>(properties); + selected.put("provider", "GCP"); + return selected; + } + + public GcsAuth getAuth() { + return auth; + } + + @Override + public Map matchedProperties() { + Map matched = new HashMap<>(super.matchedProperties()); + if (auth.getMode() != GcsAuth.Mode.HMAC) { + matched.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, auth.isAnonymous() ? "ANONYMOUS" + : auth.getNativeCredential().orElseThrow().getCredentialProviderType().name()); + matched.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, + auth.getNativeCredential().map(GcpCredential::getImpersonationServiceAccount).orElse("")); + } + return Collections.unmodifiableMap(matched); + } + + @Override + protected void customizeS3CompatibleKv(Map kv) { + if (auth.getNativeCredential().isPresent()) { + GcpCredential credential = auth.getNativeCredential().get(); + kv.remove("AWS_CREDENTIALS_PROVIDER_TYPE"); + kv.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); + putIfNotBlank(kv, GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, + credential.getImpersonationServiceAccount()); + } + } + + @Override + public Map toHadoopConfigurationMap() { + if (auth.getNativeCredential().isEmpty()) { + return super.toHadoopConfigurationMap(); + } + GcpCredential credential = auth.getNativeCredential().get(); + Map cfg = new HashMap<>(); + cfg.put("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + cfg.put("fs.gs.auth.type", credential.getCredentialProviderType() == GcpCredentialProviderType.DEFAULT + ? "APPLICATION_DEFAULT" : "COMPUTE_ENGINE"); + cfg.put("fs.gs.auth.impersonation.service.account", credential.getImpersonationServiceAccount()); + FsCacheKeys.putFsCacheKeys(cfg, this); + return Collections.unmodifiableMap(cfg); + } + @Override protected void customizeHadoopConfiguration(Map cfg) { cfg.put("fs.gs.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"); - if (!hasStaticCredentials()) { + if (auth.isAnonymous()) { cfg.put("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider"); } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index d9b11f4d3445dc..9f7f3c1e050019 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -18,14 +18,15 @@ package org.apache.doris.filesystem.gcs; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.filesystem.s3.S3CompatSignals; import org.apache.doris.filesystem.s3.S3FileSystem; import org.apache.doris.filesystem.s3.S3FileSystemProperties; -import org.apache.doris.filesystem.s3.S3ObjStorage; import org.apache.doris.filesystem.spi.FileSystemProvider; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import java.io.IOException; +import java.util.HashMap; import java.util.Map; import java.util.Set; @@ -63,9 +64,16 @@ public GcsFileSystemProperties bind(Map properties) { @Override public FileSystem create(GcsFileSystemProperties properties) throws IOException { - S3FileSystemProperties delegate = S3FileSystemProperties.of(properties.toS3CompatibleKv()); + Map delegateProperties = new HashMap<>(properties.toS3CompatibleKv()); + if (properties.getAuth().getNativeCredential().isPresent()) { + // The delegate only supplies URI/connection settings. GcsObjStorage supplies OAuth. + delegateProperties.remove(GcpCredential.CREDENTIAL_PROVIDER_TYPE); + delegateProperties.remove(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT); + delegateProperties.put("AWS_CREDENTIALS_PROVIDER_TYPE", "ANONYMOUS"); + } + S3FileSystemProperties delegate = S3FileSystemProperties.of(delegateProperties); return new S3FileSystem(delegate, - new S3ObjStorage(delegate, properties.getSupportedSchemes())); + new GcsObjStorage(delegate, properties)); } @Override diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsObjStorage.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsObjStorage.java new file mode 100644 index 00000000000000..f1f9bfb38cb25b --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsObjStorage.java @@ -0,0 +1,58 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.gcs; + +import org.apache.doris.filesystem.s3.S3FileSystemProperties; +import org.apache.doris.filesystem.s3.S3ObjStorage; + +import software.amazon.awssdk.services.s3.S3Client; + +import java.io.IOException; + +/** S3-compatible GCS client with refreshable Google OAuth credentials. */ +final class GcsObjStorage extends S3ObjStorage { + private final GcsFileSystemProperties properties; + + GcsObjStorage(S3FileSystemProperties delegate, GcsFileSystemProperties properties) { + super(delegate, properties.getSupportedSchemes()); + this.properties = properties; + } + + @Override + public String getPresignedUrl(String objectKey) throws IOException { + if (properties.getAuth().getNativeCredential().isPresent()) { + throw new UnsupportedOperationException("Presigned URLs are not supported for native GCP OAuth " + + "credentials; GCS V4 signing with IAM signBlob is required"); + } + return super.getPresignedUrl(objectKey); + } + + @Override + protected S3Client buildClient() throws IOException { + if (properties.getAuth().getNativeCredential().isEmpty()) { + return super.buildClient(); + } + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(GcsObjStorage.class.getClassLoader()); + try { + return GcpAuth.createS3Client(properties); + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java new file mode 100644 index 00000000000000..abc45ba80c8372 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java @@ -0,0 +1,92 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.gcs; + +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.GcpCredentialProviderType; + +import com.google.auth.oauth2.AccessToken; +import com.google.auth.oauth2.ComputeEngineCredentials; +import com.google.auth.oauth2.GoogleCredentials; +import com.google.auth.oauth2.ImpersonatedCredentials; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; +import software.amazon.awssdk.core.interceptor.Context; +import software.amazon.awssdk.core.interceptor.ExecutionAttributes; +import software.amazon.awssdk.core.interceptor.ExecutionInterceptor; +import software.amazon.awssdk.http.SdkHttpMethod; +import software.amazon.awssdk.http.SdkHttpRequest; + +import java.net.URI; +import java.util.Collections; +import java.util.Date; + +public class GcpAuthTest { + @Test + public void testComputeEngineCredentialsUseMetadataProviderDirectly() throws Exception { + GoogleCredentials credentials = GcpAuth.createCredentials( + new GcpCredential(GcpCredentialProviderType.COMPUTE_ENGINE, "")); + + Assertions.assertInstanceOf(ComputeEngineCredentials.class, credentials); + } + + @Test + public void testComputeEngineImpersonationWrapsMetadataCredentials() throws Exception { + GoogleCredentials credentials = GcpAuth.createCredentials( + new GcpCredential(GcpCredentialProviderType.COMPUTE_ENGINE, + "target@my-project.iam.gserviceaccount.com")); + + Assertions.assertInstanceOf(ImpersonatedCredentials.class, credentials); + } + + @Test + public void testBearerTokenInterceptorUsesCredentialRequestMetadata() { + GoogleCredentials credentials = GoogleCredentials.create( + new AccessToken("metadata-token", new Date(System.currentTimeMillis() + 3_600_000))); + ExecutionInterceptor interceptor = GcpAuth.bearerTokenInterceptor(credentials); + Context.ModifyHttpRequest context = Mockito.mock(Context.ModifyHttpRequest.class); + SdkHttpRequest request = SdkHttpRequest.builder() + .uri(URI.create("https://storage.googleapis.com/bucket/key")) + .method(SdkHttpMethod.GET) + .build(); + Mockito.when(context.httpRequest()).thenReturn(request); + + SdkHttpRequest modified = interceptor.modifyHttpRequest(context, new ExecutionAttributes()); + + Assertions.assertEquals("Bearer metadata-token", + modified.firstMatchingHeader("Authorization").orElse(null)); + } + + @Test + public void testBearerTokenInterceptorRejectsMissingAuthorizationMetadata() throws Exception { + GoogleCredentials credentials = Mockito.mock(GoogleCredentials.class); + Mockito.when(credentials.getRequestMetadata()).thenReturn(Collections.emptyMap()); + ExecutionInterceptor interceptor = GcpAuth.bearerTokenInterceptor(credentials); + Context.ModifyHttpRequest context = Mockito.mock(Context.ModifyHttpRequest.class); + SdkHttpRequest request = SdkHttpRequest.builder() + .uri(URI.create("https://storage.googleapis.com/bucket/key")) + .method(SdkHttpMethod.GET) + .build(); + Mockito.when(context.httpRequest()).thenReturn(request); + + RuntimeException exception = Assertions.assertThrows(RuntimeException.class, + () -> interceptor.modifyHttpRequest(context, new ExecutionAttributes())); + Assertions.assertTrue(exception.getMessage().contains("Failed to obtain a GCP access token")); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index d1ac47224e0fcb..ff3f78006d878b 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -87,11 +87,12 @@ void toS3CompatibleKv_delegatesHmacToCanonicalAwsKeys() { } @Test - void toS3CompatibleKv_fallsBackToAnonymousWithoutStaticCredentials() { + void toS3CompatibleKv_defaultsToAdcWithoutStaticCredentials() { Map kv = GcsFileSystemProperties.of(Map.of( "gs.endpoint", "https://storage.googleapis.com")).toS3CompatibleKv(); - Assertions.assertEquals("ANONYMOUS", kv.get("AWS_CREDENTIALS_PROVIDER_TYPE")); + Assertions.assertEquals("DEFAULT", kv.get("gs.credential_provider_type")); + Assertions.assertFalse(kv.containsKey("AWS_CREDENTIALS_PROVIDER_TYPE")); } @Test @@ -104,13 +105,13 @@ void toMap_tagsGcpProviderForBackendParity() { } @Test - void toHadoopConfigurationMap_anonymousWhenNoStaticCredentials() { + void toHadoopConfigurationMap_usesNativeAdcWithoutStaticCredentials() { Map cfg = GcsFileSystemProperties.of(Map.of( "gs.endpoint", "https://storage.googleapis.com")).toHadoopConfigurationMap(); - Assertions.assertEquals("org.apache.hadoop.fs.s3a.S3AFileSystem", cfg.get("fs.gs.impl")); - Assertions.assertEquals("org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider", - cfg.get("fs.s3a.aws.credentials.provider")); + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", cfg.get("fs.gs.impl")); + Assertions.assertEquals("APPLICATION_DEFAULT", cfg.get("fs.gs.auth.type")); + Assertions.assertFalse(cfg.containsKey("fs.s3a.aws.credentials.provider")); } @Test @@ -139,7 +140,7 @@ void of_rejectsInvalidUsePathStyle() { void of_rejectsSessionTokenWithoutStaticCredentials() { IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class, () -> GcsFileSystemProperties.of(Map.of("gs.session_token", "token"))); - Assertions.assertTrue(e.getMessage().contains("gs.session_token"), e.getMessage()); + Assertions.assertTrue(e.getMessage().contains("session token"), e.getMessage()); } @Test @@ -193,4 +194,43 @@ void region_isFixedAndNotUserConfigurable() { "gs.region", "europe-west1", "AWS_REGION", "europe-west1")).getRegion()); } + + @Test + void nativeModesPropagateToBackendHadoopAndCache() { + for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + for (String account : new String[] {"", "target@test.iam.gserviceaccount.com"}) { + GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.credential_provider_type", mode, + "gs.impersonation_service_account", account)); + Assertions.assertEquals(mode, props.toMap().get("gs.credential_provider_type")); + Assertions.assertEquals(account, props.toMap().getOrDefault("gs.impersonation_service_account", "")); + Assertions.assertFalse(props.toMap().containsKey("AWS_CREDENTIALS_PROVIDER_TYPE")); + Assertions.assertEquals(mode.equals("DEFAULT") ? "APPLICATION_DEFAULT" : mode, + props.toHadoopConfigurationMap().get("fs.gs.auth.type")); + Assertions.assertEquals(account, + props.toHadoopConfigurationMap().get("fs.gs.auth.impersonation.service.account")); + Assertions.assertEquals(mode, props.matchedProperties().get("gs.credential_provider_type")); + Assertions.assertEquals(account, props.matchedProperties().get("gs.impersonation_service_account")); + } + } + } + + @Test + void explicitAnonymousUsesUnsignedBackendAndHadoop() { + GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.credential_provider_type", "ANONYMOUS")); + Assertions.assertEquals("ANONYMOUS", props.toMap().get("AWS_CREDENTIALS_PROVIDER_TYPE")); + Assertions.assertFalse(props.toMap().containsKey("gs.credential_provider_type")); + Assertions.assertEquals("org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider", + props.toHadoopConfigurationMap().get("fs.s3a.aws.credentials.provider")); + } + + @Test + void inferredAdcAndExplicitAdcHaveSameAuthenticationCacheIdentity() { + GcsFileSystemProperties implicit = GcsFileSystemProperties.of(Map.of("provider", "GCP")); + GcsFileSystemProperties explicit = GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.credential_provider_type", "DEFAULT")); + Assertions.assertEquals(implicit.matchedProperties(), explicit.matchedProperties()); + Assertions.assertEquals(implicit.toMap(), explicit.toMap()); + } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java index ce8f2049423ce9..6a08156b9ace4c 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java @@ -188,4 +188,15 @@ void sensitivePropertyKeys_coverSecretAliasesButNotAccessKey() { void name_isGcs() { Assertions.assertEquals("GCS", provider.name()); } + + @Test + void nativeOAuthRejectsAwsPresigningBeforeCredentialLoading() { + GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "s3.bucket", "bucket")); + GcsObjStorage storage = new GcsObjStorage( + org.apache.doris.filesystem.s3.S3FileSystemProperties.of(Map.of( + "s3.endpoint", "storage.googleapis.com", "s3.region", "us-east1", + "AWS_CREDENTIALS_PROVIDER_TYPE", "ANONYMOUS")), props); + Assertions.assertThrows(UnsupportedOperationException.class, () -> storage.getPresignedUrl("key")); + } } diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java index b78a71759d930c..9f4ba05db6a2ee 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java @@ -180,11 +180,7 @@ public static boolean guessAllowed(Map properties) { * key comparison is genuinely case-insensitive, which is what legacy clearly intended. */ public static boolean guessIsGcs(Map properties) { - if (StringUtils.isNotBlank(properties.get(GCS_ENDPOINT_KEY))) { - return true; - } - String endpoint = endpointForGuessing(properties); - return endpoint != null && endpoint.toLowerCase(Locale.ROOT).endsWith(GCS_ENDPOINT_SUFFIX); + return org.apache.doris.filesystem.auth.GcsAuthResolver.guessIsGcs(properties); } /** diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java index dc6ca844360cf0..ecb5731d2718ea 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java @@ -208,6 +208,10 @@ public static S3FileSystemProperties of(Map properties) { @Override public void validate() { + if (org.apache.doris.filesystem.auth.GcsAuthResolver.hasNativeCredentialProperties(rawProperties)) { + org.apache.doris.filesystem.auth.GcsAuthResolver.resolve(rawProperties); + throw new IllegalArgumentException("Native GCP authentication cannot be used with S3 properties."); + } new ParamRules() .requireTogether(new String[] {accessKey, secretKey}, "s3.access_key and s3.secret_key must be set together") diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java index a081233add1dbf..190abbd1fb320a 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java @@ -401,4 +401,10 @@ void forceParsingByStandardUri_parsesS3SchemeAsStandardUrl() { Assertions.assertEquals("s3.us-west-2.amazonaws.com", properties.getEndpoint()); Assertions.assertEquals("us-west-2", properties.getRegion()); } + + @Test + void rejectsNativeGcpCredentialsInsteadOfFallingBackToAws() { + Assertions.assertThrows(IllegalArgumentException.class, () -> S3FileSystemProperties.of(Map.of( + "provider", "S3", "s3.region", "us-east-1", "gs.credential_provider_type", "DEFAULT"))); + } } diff --git a/fe/pom.xml b/fe/pom.xml index a04b3ad9838436..6f3fea09d03892 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -390,6 +390,7 @@ under the License. 3.0.9 42.7.13 3.4.2 + 3.1.18 1.8 1.2.0 1.5.0 @@ -1457,6 +1458,19 @@ under the License. + + + com.google.cloud.bigdataoss + gcs-connector + ${gcs-connector.version} + shaded + + + * + * + + + org.apache.hadoop hadoop-azure diff --git a/gensrc/proto/cloud.proto b/gensrc/proto/cloud.proto index afb7d41a8647b4..1bc6d4b69341a1 100644 --- a/gensrc/proto/cloud.proto +++ b/gensrc/proto/cloud.proto @@ -287,6 +287,24 @@ enum CredProviderTypePB { ANONYMOUS = 8; // AnonymousAWSCredentialsProvider } +message GcpCredentialPB { + enum CredentialProviderType { + DEFAULT = 0; + COMPUTE_ENGINE = 1; + } + + optional CredentialProviderType credential_provider_type = 1; + optional string impersonation_service_account = 2; +} + +// Provider credentials are grouped in a regular message so older readers can +// still observe that a credential envelope is present when a new provider is +// added. Validation must reject multiple provider credentials and ensure the +// selected credential matches ObjectStoreInfoPB.provider. +message ObjectStoreCredentialPB { + optional GcpCredentialPB gcp_credential = 1; +} + message ObjectStoreInfoPB { // presigned url use // oss,aws,cos,obs,bos @@ -320,6 +338,8 @@ message ObjectStoreInfoPB { optional CredProviderTypePB cred_provider_type = 17; optional string role_arn = 18; // aws assumed role's arn optional string external_id = 19; // aws assumed role's external_id if configure + + optional ObjectStoreCredentialPB credential = 20; } // The legacy ObjectStoreInfoPB is stored in InstanceInfoPB diff --git a/gensrc/thrift/AgentService.thrift b/gensrc/thrift/AgentService.thrift index 5ca0f531c06948..e769f769a9966b 100644 --- a/gensrc/thrift/AgentService.thrift +++ b/gensrc/thrift/AgentService.thrift @@ -110,6 +110,23 @@ enum TCredProviderType { ANONYMOUS = 7 // AnonymousAWSCredentialsProvider } +enum TGcpCredentialProviderType { + DEFAULT = 0, + COMPUTE_ENGINE = 1 +} + +struct TGcpCredential { + 1: optional TGcpCredentialProviderType credential_provider_type + 2: optional string impersonation_service_account +} + +// Keep this struct aligned with ObjectStoreCredentialPB. Append new fields with +// new IDs; validation rejects multiple provider credentials and ensures the +// selected credential matches TS3StorageParam.provider. +struct TCredential { + 1: optional TGcpCredential gcp_credential +} + struct TS3StorageParam { 1: optional string endpoint 2: optional string region @@ -127,6 +144,7 @@ struct TS3StorageParam { 13: optional TCredProviderType cred_provider_type 14: optional string role_arn // aws assumed role's arn 15: optional string external_id // aws assumed role's external_id if configure + 16: optional TCredential credential } struct TStoragePolicy { diff --git a/regression-test/conf/regression-conf.groovy b/regression-test/conf/regression-conf.groovy index 8240d3f082e094..a21b18c08588e1 100644 --- a/regression-test/conf/regression-conf.groovy +++ b/regression-test/conf/regression-conf.groovy @@ -325,6 +325,28 @@ anymousS3Region="eu-west-3" anymousS3ExpectDataCount="8365" awsInstanceProfileRegion="us-east-1" +// Object storage IAM regression tests run one provider per invocation: AWS or GCP. +objectStorageIamProvider="" +objectStorageIamEndpoint="" +objectStorageIamRegion="" +objectStorageIamBucket="" +objectStorageIamPrefix="regression/object_storage_iam_p0" +objectStorageIamDataPath="regression/tpch/sf0.01/customer.csv.gz" +// AWS authentication. +objectStorageIamAwsRoleArn="" +objectStorageIamAwsExternalId="" +// GCP authentication. Credential provider types are comma-separated DEFAULT and/or +// COMPUTE_ENGINE. DEFAULT also tests omitted authentication properties; leaving both GCP +// settings empty tests only omitted properties. A non-empty account adds COMPUTE_ENGINE +// impersonation. The IAM data fixture must be private so anonymous reads can be rejected. +objectStorageIamGcpCredentialProviderTypes="" +objectStorageIamGcpImpersonationServiceAccount="" +// Optional public GCS fixture for anonymous TVF reads: gzip CSV with a '|' separator. +// Configure a gs:// URI and its positive row count to enable test_gcp_tvf_anonymous. +// This public fixture must differ from the private objectStorageIamDataPath fixture. +objectStorageIamGcpAnonymousUri="" +objectStorageIamGcpAnonymousExpectedRows="" + // hudi p0 external regression test config // To enable hudi test, you need first start hudi container. // See `docker/thirdparties/run-thirdparties-docker.sh -c hudi` diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy index a87744a86a387e..69ab664ec2e6b0 100644 --- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy +++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy @@ -1027,25 +1027,15 @@ class Syncer { """ } - void createS3RepositoryWithRole(String name, boolean readOnly = false) { - String roleArn = suite.context.config.awsRoleArn - String externalId = suite.context.config.awsExternalId - String endpoint = suite.context.config.awsEndpoint - String region = suite.context.config.awsRegion - String bucket = suite.context.config.awsBucket - String prefix = suite.context.config.awsPrefix - + void createObjectStorageIamRepository(String name, Map iamConfig, Map authCase, boolean readOnly = false) { suite.try_sql "DROP REPOSITORY `${name}`" suite.sql """ CREATE ${readOnly ? "READ ONLY" : ""} REPOSITORY `${name}` WITH S3 - ON LOCATION "s3://${bucket}/${prefix}/aws_iam_role_p0/${name}" + ON LOCATION "${iamConfig.scheme}://${iamConfig.bucket}/${iamConfig.prefix}/${name}" PROPERTIES ( - "s3.endpoint" = "${endpoint}", - "s3.region" = "${region}", - "s3.role_arn" = "${roleArn}", - "s3.external_id" = "${externalId}" + ${authCase.storageSqlProperties} ) """ } diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/ObjectStorageIamTestUtils.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/ObjectStorageIamTestUtils.groovy new file mode 100644 index 00000000000000..e3e94fbf3662d2 --- /dev/null +++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/ObjectStorageIamTestUtils.groovy @@ -0,0 +1,193 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.regression.util + +import java.util.Locale + +/** + * Builds the IAM authentication matrix shared by the object_storage_iam_p0 suites. + * + *

One regression-test invocation selects exactly one cloud with + * {@code objectStorageIamProvider=AWS} or {@code objectStorageIamProvider=GCP}. Common connection + * settings are reused by both providers. Leaving {@code objectStorageIamProvider} empty disables + * the shared cases; AWS-only suites also use {@link #isProvider(Map, String)} to skip themselves + * during a GCP run. + * + *

AWS role example in regression-conf.groovy: + * 

+ * objectStorageIamProvider="AWS"
+ * objectStorageIamEndpoint="https://s3.us-east-1.amazonaws.com"
+ * objectStorageIamRegion="us-east-1"
+ * objectStorageIamBucket="my-bucket"
+ * objectStorageIamPrefix="regression/object_storage_iam_p0"
+ * objectStorageIamDataPath="regression/tpch/sf0.01/customer.csv.gz"
+ * objectStorageIamAwsRoleArn="arn:aws:iam::123456789012:role/my-role"
+ * objectStorageIamAwsExternalId="optional-external-id"
+ *
+ * + *

GCP example covering DEFAULT, COMPUTE_ENGINE, and service-account impersonation: + * 

+ * objectStorageIamProvider="GCP"
+ * objectStorageIamEndpoint="https://storage.googleapis.com"
+ * objectStorageIamRegion="us-east1"
+ * objectStorageIamBucket="my-bucket"
+ * objectStorageIamPrefix="regression/object_storage_iam_p0"
+ * objectStorageIamDataPath="regression/tpch/sf0.01/customer.csv.gz"
+ * objectStorageIamGcpCredentialProviderTypes="DEFAULT,COMPUTE_ENGINE"
+ * objectStorageIamGcpImpersonationServiceAccount="target@project.iam.gserviceaccount.com"
+ *
+ * + *

The comma-separated GCP provider list controls which direct credential cases run. DEFAULT also + * tests omitted authentication properties against the same private fixture. If both the provider + * list and impersonation account are empty, only the omitted-authentication case runs. A non-empty + * impersonation service account adds one more COMPUTE_ENGINE case. Each shared suite iterates over + * the resulting {@code authCases}. The test selector uses AWS/GCP, while SQL properties use the + * public S3/GCP provider values. + */ +class ObjectStorageIamTestUtils { + private static final List PROVIDERS = ["AWS", "GCP"] + private static final List GCP_CREDENTIAL_PROVIDER_TYPES = ["DEFAULT", "COMPUTE_ENGINE"] + private static final List REQUIRED_CONFIGS = [ + "objectStorageIamEndpoint", + "objectStorageIamRegion", + "objectStorageIamBucket", + "objectStorageIamPrefix", + "objectStorageIamDataPath" + ] + + static Map getConfig(Map configs) { + String provider = value(configs, "objectStorageIamProvider")?.toUpperCase(Locale.ROOT) + if (provider == null || provider.isEmpty()) { + return null + } + if (!PROVIDERS.contains(provider)) { + throw new IllegalArgumentException("Unsupported objectStorageIamProvider: ${provider}; " + + "supported values are ${PROVIDERS}") + } + + List missingConfigs = REQUIRED_CONFIGS.findAll { isBlank(value(configs, it)) } + if (!missingConfigs.isEmpty()) { + throw new IllegalArgumentException("Missing object storage IAM configs: ${missingConfigs}") + } + + List authCases = provider == "AWS" + ? getAwsAuthCases(configs) + : getGcpAuthCases(configs) + String endpointProperty = provider == "AWS" ? "s3.endpoint" : "gs.endpoint" + String propertyProvider = provider == "AWS" ? "S3" : "GCP" + String endpoint = value(configs, "objectStorageIamEndpoint") + String region = value(configs, "objectStorageIamRegion") + authCases.each { authCase -> + Map storageProperties = new LinkedHashMap<>() + storageProperties.put("provider", propertyProvider) + storageProperties.put(endpointProperty, endpoint) + storageProperties.put("s3.region", region) + storageProperties.putAll(authCase.properties) + authCase.storageProperties = storageProperties + authCase.storageSqlProperties = toSqlProperties(storageProperties) + } + return [ + provider: provider, + propertyProvider: propertyProvider, + scheme: provider == "AWS" ? "s3" : "gs", + endpoint: endpoint, + region: region, + bucket: value(configs, "objectStorageIamBucket"), + prefix: value(configs, "objectStorageIamPrefix"), + dataPath: value(configs, "objectStorageIamDataPath"), + roleArn: value(configs, "objectStorageIamAwsRoleArn"), + externalId: value(configs, "objectStorageIamAwsExternalId"), + authCases: authCases + ] + } + + static boolean isProvider(Map configs, String expectedProvider) { + String provider = value(configs, "objectStorageIamProvider") + return provider != null && provider.equalsIgnoreCase(expectedProvider) + } + + private static List getAwsAuthCases(Map configs) { + String roleArn = value(configs, "objectStorageIamAwsRoleArn") + if (roleArn == null || roleArn.isEmpty()) { + throw new IllegalArgumentException("objectStorageIamAwsRoleArn must be configured for AWS") + } + + Map properties = ["s3.role_arn": roleArn] + String externalId = value(configs, "objectStorageIamAwsExternalId") + if (externalId != null && !externalId.isEmpty()) { + properties.put("s3.external_id", externalId) + } + return [authCase("aws_role", properties)] + } + + private static List getGcpAuthCases(Map configs) { + String configuredTypes = value(configs, "objectStorageIamGcpCredentialProviderTypes") + List providerTypes = (configuredTypes ?: "").split(",") + .collect { it.trim().toUpperCase(Locale.ROOT) } + .findAll { !it.isEmpty() } + .unique() + List unsupportedTypes = providerTypes.findAll { + !GCP_CREDENTIAL_PROVIDER_TYPES.contains(it) + } + if (!unsupportedTypes.isEmpty()) { + throw new IllegalArgumentException( + "Unsupported objectStorageIamGcpCredentialProviderTypes: ${unsupportedTypes}; " + + "supported values are ${GCP_CREDENTIAL_PROVIDER_TYPES}") + } + + List cases = providerTypes.collect { providerType -> + authCase("gcp_${providerType.toLowerCase(Locale.ROOT)}", + ["gs.credential_provider_type": providerType]) + } + String serviceAccount = value(configs, "objectStorageIamGcpImpersonationServiceAccount") + if (serviceAccount != null && !serviceAccount.isEmpty()) { + cases.add(authCase("gcp_impersonation", [ + "gs.credential_provider_type": "COMPUTE_ENGINE", + "gs.impersonation_service_account": serviceAccount + ])) + } + if (providerTypes.contains("DEFAULT") || cases.isEmpty()) { + // Exercise the VM-attached service account without either native authentication property. + // Shared TVF, Load, Outfile, Export, Resource, Catalog and Vault suites all consume this case. + cases.add(0, authCase("gcp_default_omitted", [:])) + } + return cases + } + + private static Map authCase(String name, Map properties) { + return [ + name: name, + properties: properties, + authProperties: toSqlProperties(properties) + ] + } + + static String toSqlProperties(Map properties) { + return properties.collect { key, propertyValue -> + "\"${key}\" = \"${propertyValue}\"" + }.join(",\n") + } + + private static String value(Map configs, String key) { + return configs.get(key)?.toString()?.trim() + } + + private static boolean isBlank(String value) { + return value == null || value.isEmpty() + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy b/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy new file mode 100644 index 00000000000000..28793ec085e9c9 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy @@ -0,0 +1,234 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import groovy.json.JsonSlurper +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_alter_resource_with_role") { + if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { + logger.info("skip ${name} because objectStorageIamProvider is not AWS") + return + } + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + + + if (isCloudMode()) { + logger.info("skip ${name} case, because it is cloud mode") + return + } + + def tableName = "test_alter_resource_with_role" + def randomStr = UUID.randomUUID().toString().replace("-", "") + def resourceName = "alter_resource_${randomStr}" + def policyName = "alter_policy_${randomStr}" + + def awsEndpoint = config.endpoint + def region = config.region + def bucket = config.bucket + def roleArn = config.roleArn + def externalId = config.externalId ?: "" + def prefix = config.prefix + + def awsAccessKey = context.config.awsAccessKey + def awsSecretKey = context.config.awsSecretKey + + sql """ + CREATE RESOURCE IF NOT EXISTS "${resourceName}" + PROPERTIES( + "type"="s3", + "AWS_ENDPOINT" = "${awsEndpoint}", + "AWS_REGION" = "${region}", + "AWS_BUCKET" = "${bucket}", + "AWS_ROOT_PATH" = "${prefix}/test_alter_resource_with_role/${randomStr}", + "AWS_ACCESS_KEY" = "error_ak", + "AWS_SECRET_KEY" = "error_sk", + "s3_validity_check" = "false" + ); + """ + + sql """ + CREATE STORAGE POLICY IF NOT EXISTS ${policyName} + PROPERTIES( + "storage_resource" = "${resourceName}", + "cooldown_ttl" = "1" + ) + """ + + sql """ + DROP TABLE IF EXISTS ${tableName} FORCE; + """ + + sql """ + CREATE TABLE ${tableName} + ( + siteid INT DEFAULT '10', + citycode SMALLINT NOT NULL, + username VARCHAR(32) DEFAULT '', + pv BIGINT SUM DEFAULT '0' + ) + AGGREGATE KEY(siteid, citycode, username) + DISTRIBUTED BY HASH(siteid) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1", + "storage_policy" = "${policyName}" + ) + """ + + sql """insert into ${tableName}(siteid, citycode, username, pv) values (1, 1, "xxx", 1), + (2, 2, "yyy", 2), + (3, 3, "zzz", 3) + """ + + def result = sql """ SHOW RESOURCES WHERE NAME = "${resourceName}"; """ + log.info("result:${result}") + assertTrue(!result.toString().contains(roleArn)) + assertTrue(!result.toString().contains(externalId)); + assertTrue(result.toString().contains("error_ak")); + + // data_sizes is one arrayList, t is tablet + def fetchDataSize = {List data_sizes, Map t -> + def tabletId = t.TabletId + def meta_url = t.MetaUrl + def clos = { respCode, body -> + logger.info("test ttl expired resp Code {}", "${respCode}".toString()) + assertEquals("${respCode}".toString(), "200") + String out = "${body}".toString() + logger.info("test fetchBeHttp, resp body: ${out}") + def obj = new JsonSlurper().parseText(out) + data_sizes[0] = obj.local_data_size + data_sizes[1] = obj.remote_data_size + } + meta_url = meta_url.replace("header", "data_size") + + def i = meta_url.indexOf("/api") + def endPoint = meta_url.substring(0, i) + def metaUri = meta_url.substring(i) + logger.info("test fetchBeHttp, endpoint:${endPoint}, metaUri:${metaUri}") + i = endPoint.lastIndexOf('/') + endPoint = endPoint.substring(i + 1) + + httpTest { + endpoint {endPoint} + uri metaUri + op "get" + check clos + } + } + + List sizes = [-1, -1] + def tablets = sql_return_maparray """ + SHOW TABLETS FROM ${tableName} + """ + log.info( "test tablets not empty:${tablets}") + fetchDataSize(sizes, tablets[0]) + def retry = 12 + while (sizes[1] == 0 && retry-- > 0) { + log.info( "test remote size is zero, sleep 10s") + sleep(10000) + tablets = sql_return_maparray """ + SHOW TABLETS FROM ${tableName} + """ + fetchDataSize(sizes, tablets[0]) + } + assertTrue(sizes[1] == 0, "remote size expected is zero, but got ${sizes[1]}") + assertTrue(tablets.size() > 0) + + // alter resource to arn role + sql """ truncate table ${tableName} """ + sql """ALTER RESOURCE "${resourceName}" + PROPERTIES( + "AWS_ROLE_ARN" = "${roleArn}", + "AWS_EXTERNAL_ID" = "${externalId}" + );""" + + result = sql """ SHOW RESOURCES WHERE NAME = "${resourceName}"; """ + log.info("result:${result}") + assertTrue(result.toString().contains(roleArn)) + assertTrue(result.toString().contains(externalId)); + assertTrue(!result.toString().contains("error_ak")); + + // report_tablet_interval_seconds = 60 sleep wait for report policy + sleep(120 * 1000) + + sql """insert into ${tableName}(siteid, citycode, username, pv) values (1, 1, "xxx", 1), + (4, 4, "yyy", 5), + (5, 5, "zzz", 5), + (6, 6, "xxx", 6) + """ + + sizes = [-1, -1] + tablets = sql_return_maparray """ + SHOW TABLETS FROM ${tableName} + """ + log.info( "test tablets not empty:${tablets}") + fetchDataSize(sizes, tablets[0]) + retry = 100 + while (sizes[1] == 0 && retry-- > 0) { + log.info( "test remote size is zero, sleep 10s") + sleep(10000) + tablets = sql_return_maparray """ + SHOW TABLETS FROM ${tableName} + """ + fetchDataSize(sizes, tablets[0]) + } + assertTrue(sizes[1] != 0, "remote size is still zero, maybe some error occurred") + assertTrue(tablets.size() > 0) + log.info( "test remote size not zero") + + // alter resource to correct ak/sk + sql """ truncate table ${tableName} """ + sql """ALTER RESOURCE "${resourceName}" + PROPERTIES( + "AWS_ACCESS_KEY" = "${awsAccessKey}", + "AWS_SECRET_KEY" = "${awsSecretKey}" + );""" + + result = sql """ SHOW RESOURCES WHERE NAME = "${resourceName}"; """ + log.info("result:${result}") + assertTrue(!result.toString().contains(roleArn)) + assertTrue(!result.toString().contains(externalId)); + assertTrue(result.toString().contains(awsAccessKey)); + + // report_tablet_interval_seconds = 60 sleep wait for report policy + sleep(120 * 1000) + + sql """insert into ${tableName}(siteid, citycode, username, pv) values (1, 1, "xxx", 1), + (4, 4, "yyy", 5), + (5, 5, "zzz", 5), + (6, 6, "xxx", 6), + (7, 7, "aaa", 8) + """ + + sizes = [-1, -1] + tablets = sql_return_maparray """ + SHOW TABLETS FROM ${tableName} + """ + log.info( "test tablets not empty:${tablets}") + fetchDataSize(sizes, tablets[0]) + retry = 100 + while (sizes[1] == 0 && retry-- > 0) { + log.info( "test remote size is zero, sleep 10s") + sleep(10000) + tablets = sql_return_maparray """ + SHOW TABLETS FROM ${tableName} + """ + fetchDataSize(sizes, tablets[0]) + } + assertTrue(sizes[1] != 0, "remote size is still zero, maybe some error occurred") + assertTrue(tablets.size() > 0) + log.info( "test remote size not zero") +} diff --git a/regression-test/suites/object_storage_iam_p0/test_alter_storage_vault_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_alter_storage_vault_iam.groovy new file mode 100644 index 00000000000000..b6c80a6b0c9733 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_alter_storage_vault_iam.groovy @@ -0,0 +1,158 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_alter_storage_vault_iam") { + if (!isCloudMode()) { + logger.info("skip ${name} case, because not cloud mode") + return + } + + if (!enableStoragevault()) { + logger.info("skip ${name} case, because storage vault not enabled") + return + } + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def randomStr = UUID.randomUUID().toString().replace("-", "") + def s3VaultName = "object_storage_iam_" + randomStr + def firstAuthCase = config.authCases[0] + def awsAccessKey = context.config.awsAccessKey + def awsSecretKey = context.config.awsSecretKey + + sql """ + CREATE STORAGE VAULT IF NOT EXISTS ${s3VaultName} + PROPERTIES ( + "type"="S3", + ${firstAuthCase.storageSqlProperties}, + "s3.root.path" = "${config.prefix}/test_alter_storage_vault_iam/${s3VaultName}", + "s3.bucket" = "${config.bucket}", + "s3.external_endpoint" = "", + "use_path_style" = "false" + ); + """ + + sql """ + CREATE TABLE ${s3VaultName} ( + C_CUSTKEY INTEGER NOT NULL, + C_NAME INTEGER NOT NULL + ) + DUPLICATE KEY(C_CUSTKEY, C_NAME) + DISTRIBUTED BY HASH(C_CUSTKEY) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1", + "storage_vault_name" = ${s3VaultName} + ) + """ + sql """ insert into ${s3VaultName} values(1, 1); """ + sql """ sync;""" + def result = sql """ select * from ${s3VaultName}; """ + assertEquals(result.size(), 1); + + def assertVaultProperties = { Map expected, List unexpected -> + def vaultInfos = sql "SHOW STORAGE VAULTS" + def vaultInfo = vaultInfos.find { it[0].equals(s3VaultName) } + assertTrue(vaultInfo != null, "storage vault ${s3VaultName} was not found") + def newProperties = vaultInfo[2] + logger.info("newProperties: ${newProperties}") + expected.each { key, value -> + assertTrue(newProperties.contains(value), "${key} was not updated") + } + unexpected.findAll { it != null && !it.isEmpty() }.each { value -> + assertFalse(newProperties.contains(value), "stale credential property was not removed") + } + } + + def insertAndAssert = { int rowCount -> + sql "insert into ${s3VaultName} values(${rowCount}, ${rowCount})" + sql "sync" + result = sql "select * from ${s3VaultName}" + assertEquals(result.size(), rowCount) + } + + if (config.provider == "AWS") { + sql """ + ALTER STORAGE VAULT ${s3VaultName} + PROPERTIES ( + "type"="S3", + "s3.access_key" = "${awsAccessKey}", + "s3.secret_key" = "${awsSecretKey}" + ); + """ + assertVaultProperties(["s3.access_key": awsAccessKey], [config.roleArn]) + insertAndAssert(2) + + sql """ + ALTER STORAGE VAULT ${s3VaultName} + PROPERTIES ( + "type"="S3", + ${firstAuthCase.authProperties} + ); + """ + assertVaultProperties(firstAuthCase.properties, [awsAccessKey]) + insertAndAssert(3) + } else { + def impersonationAccounts = config.authCases.collect { + it.properties.get("gs.impersonation_service_account") + }.findAll { it != null } + // Omitted authentication is a CREATE default, not a request to reset credentials on ALTER. + // Exercise a storage-only update before switching to the explicitly configured providers. + sql """ + ALTER STORAGE VAULT ${s3VaultName} PROPERTIES ( + "type" = "S3", + "use_path_style" = "true" + ) + """ + assertVaultProperties(firstAuthCase.properties ?: ["gs.credential_provider_type": "DEFAULT"], []) + insertAndAssert(2) + + def explicitCases = config.authCases.findAll { !it.properties.isEmpty() } + def transitionCases = explicitCases.isEmpty() ? [] : explicitCases + [explicitCases[0]] + transitionCases.eachWithIndex { authCase, index -> + logger.info("alter ${name} to ${authCase.name}") + def alterProperties = new LinkedHashMap(authCase.properties) + // A direct-credential transition must explicitly clear the old impersonation target. + alterProperties.putIfAbsent("gs.impersonation_service_account", "") + sql """ + ALTER STORAGE VAULT ${s3VaultName} + PROPERTIES ( + "type"="S3", + ${ObjectStorageIamTestUtils.toSqlProperties(alterProperties)} + ); + """ + def unexpected = authCase.properties.containsKey("gs.impersonation_service_account") + ? [] : impersonationAccounts + assertVaultProperties(authCase.properties, unexpected) + insertAndAssert(index * 2 + 3) + + // Unrelated ALTERs must preserve COMPUTE_ENGINE and impersonation as well as DEFAULT. + sql """ + ALTER STORAGE VAULT ${s3VaultName} PROPERTIES ( + "type" = "S3", + "use_path_style" = "${index % 2 == 0 ? 'false' : 'true'}" + ) + """ + assertVaultProperties(authCase.properties, unexpected) + insertAndAssert(index * 2 + 4) + } + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_backup_restore_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_backup_restore_iam.groovy new file mode 100644 index 00000000000000..1d14c95705409c --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_backup_restore_iam.groovy @@ -0,0 +1,101 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_backup_restore_iam") { + if (isCloudMode()) { + logger.info("skip ${name} case, because cloud mode not support") + return + } + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + String suiteName = "test_backup_restore_iam" + + String dbName = "${suiteName}_db" + String tableName = "${suiteName}_table" + + def syncer = getSyncer() + + sql "CREATE DATABASE IF NOT EXISTS ${dbName}" + sql "DROP TABLE IF EXISTS ${dbName}.${tableName} FORCE;" + sql """ + CREATE TABLE ${dbName}.${tableName} ( + `id` LARGEINT NOT NULL, + `count` LARGEINT SUM DEFAULT "0") + AGGREGATE KEY(`id`) + DISTRIBUTED BY HASH(`id`) BUCKETS 2 + PROPERTIES + ( + "replication_num" = "1" + ) + """ + + List values = [] + for (int i = 1; i <= 10; ++i) { + values.add("(${i}, ${i})") + } + sql "INSERT INTO ${dbName}.${tableName} VALUES ${values.join(",")}" + def result = sql "SELECT * FROM ${dbName}.${tableName}" + assertEquals(result.size(), values.size()); + + config.authCases.each { authCase -> + String suffix = UUID.randomUUID().toString().replace("-", "") + String repoName = "obj_iam_${authCase.name}_${suffix}" + String snapshotName = "obj_iam_${authCase.name}_${suffix.substring(0, 8)}" + + logger.info("run ${name} with ${authCase.name}") + syncer.createObjectStorageIamRepository(repoName, config, authCase) + + sql """ + BACKUP SNAPSHOT ${dbName}.${snapshotName} + TO `${repoName}` + ON (${tableName}) + """ + + syncer.waitSnapshotFinish(dbName) + + def snapshot = syncer.getSnapshotTimestamp(repoName, snapshotName) + assertTrue(snapshot != null) + + sql "TRUNCATE TABLE ${dbName}.${tableName}" + + sql """ + RESTORE SNAPSHOT ${dbName}.${snapshotName} + FROM `${repoName}` + ON (`${tableName}`) + PROPERTIES + ( + "backup_timestamp" = "${snapshot}", + "reserve_replica" = "true" + ) + """ + + syncer.waitAllRestoreFinish(dbName) + + result = sql "SELECT * FROM ${dbName}.${tableName}" + assertEquals(values.size(), result.size(), authCase.name) + sql "DROP REPOSITORY `${repoName}`" + } + + sql "DROP TABLE ${dbName}.${tableName} FORCE" + sql "DROP DATABASE ${dbName} FORCE" +} diff --git a/regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy b/regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy new file mode 100644 index 00000000000000..a617fe34ffbb24 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy @@ -0,0 +1,134 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import com.google.common.base.Strings; +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_catalog_instance_profile") { + if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { + logger.info("skip ${name} because objectStorageIamProvider is not AWS") + return + } + + if (Strings.isNullOrEmpty(context.config.otherConfigs.get("hiveGlueInstanceProfileQueryTableName"))) { + return + } + sql """ ADMIN SET FRONTEND CONFIG ("aws_credentials_provider_version"="v2"); """ + String hiveGlueQueryTableName = context.config.otherConfigs.get("hiveGlueInstanceProfileQueryTableName") + String hiveGlueExpectCounts = context.config.otherConfigs.get("hiveGlueInstanceProfileExpectCounts") + String icebergFsQueryTableName = context.config.otherConfigs.get("icebergFsInstanceProfileQueryTableName") + String icebergFsExpectCounts = context.config.otherConfigs.get("icebergFsInstanceProfileExpectCounts") + String icebergFsWarehouse = context.config.otherConfigs.get("icebergFsInstanceProfileWarehouse") + String region = context.config.otherConfigs.get("awsInstanceProfileRegion") + //query method + def createCatalogAndQuery = { catalogProps, catalogName, queryTableName, expectCounts -> + sql """drop catalog if exists ${catalogName}""" + sql """ + ${catalogProps} + """ + def result = sql """ + select count(1) from ${catalogName}.${queryTableName}; + """ + println("result: ${result}") + def countValue = result[0][0] + assertTrue(countValue == expectCounts.toInteger()) + sql """drop catalog if exists ${catalogName}""" + } + def assertCatalogAndQueryException = { catalogProps, catalogName, queryTableName -> + sql """drop catalog if exists ${catalogName}""" + sql """ + ${catalogProps} + """ + boolean failed = false + try { + sql """ + switch ${catalogName}; + """ + sql """ + show databases; + """ + sql """ + select count(1) from ${catalogName}.${queryTableName}; + """ + }catch (Exception e){ + failed = true + } + assertTrue(failed) + } + String hiveGlueCatalogProps = """ + create catalog hive_glue_catalog_instance_profile properties( + "type"="hms", + "hive.metastore.type"="glue", + "glue.region"="${region}", + "glue.endpoint" = "https://glue.${region}.amazonaws.com" + ); + """ + createCatalogAndQuery(hiveGlueCatalogProps, "hive_glue_catalog_instance_profile", hiveGlueQueryTableName, hiveGlueExpectCounts) + hiveGlueCatalogProps = """ + create catalog hive_glue_catalog_instance_profile properties( + "type"="hms", + "hive.metastore.type"="glue", + "glue.credentials_provider_type"="INSTANCE_PROFILE", + "glue.region"="${region}", + "glue.endpoint" = "https://glue.${region}.amazonaws.com" + ); + """ + createCatalogAndQuery(hiveGlueCatalogProps, "hive_glue_catalog_instance_profile", hiveGlueQueryTableName, hiveGlueExpectCounts) + hiveGlueCatalogProps = """ + create catalog hive_glue_catalog_instance_profile properties( + "type"="hms", + "hive.metastore.type"="glue", + "glue.credentials_provider_type"="CONTAINER", + "glue.region"="${region}", + "glue.endpoint" = "https://glue.${region}.amazonaws.com" + ); + """ + assertCatalogAndQueryException(hiveGlueCatalogProps,"hive_glue_catalog_instance_profile", hiveGlueQueryTableName) + String icebergFsCatalogProps = """ + create catalog iceberg_fs_catalog_instance_profile properties( + "type"="iceberg", + "warehouse"="${icebergFsWarehouse}", + "iceberg.catalog.type"="hadoop", + "s3.region" = "${region}", + "s3.endpoint" = "https://s3.${region}.amazonaws.com" + ); + """ + createCatalogAndQuery(icebergFsCatalogProps, "iceberg_fs_catalog_instance_profile", icebergFsQueryTableName, icebergFsExpectCounts) + icebergFsCatalogProps = """ + create catalog iceberg_fs_catalog_instance_profile properties( + "type"="iceberg", + "warehouse"="${icebergFsWarehouse}", + "iceberg.catalog.type"="hadoop", + "s3.credentials_provider_type"="INSTANCE_PROFILE", + "s3.region" = "${region}", + "s3.endpoint" = "https://s3.${region}.amazonaws.com" + ); + """ + createCatalogAndQuery(icebergFsCatalogProps, "iceberg_fs_catalog_instance_profile", icebergFsQueryTableName, icebergFsExpectCounts) + icebergFsCatalogProps = """ + create catalog iceberg_fs_catalog_instance_profile properties( + "type"="iceberg", + "warehouse"="${icebergFsWarehouse}", + "iceberg.catalog.type"="hadoop", + "s3.credentials_provider_type"="CONTAINER", + "s3.region" = "${region}", + "s3.endpoint" = "https://s3.${region}.amazonaws.com" + ); + """ + assertCatalogAndQueryException(icebergFsCatalogProps,"iceberg_fs_catalog_instance_profile",icebergFsQueryTableName) + +} diff --git a/regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy b/regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy new file mode 100644 index 00000000000000..d6eda63c8ef3b0 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import com.google.common.base.Strings; +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_catalog_with_role") { + if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { + logger.info("skip ${name} because objectStorageIamProvider is not AWS") + return + } + if (Strings.isNullOrEmpty(context.config.otherConfigs.get("awsLakesRoleArn"))) { + logger.info("skip ${name} case, because awsRoleArn is null or empty") + return + } + //query method + def createCatalogAndQuery = { catalogProps, catalogName, queryTableName, expectCounts -> + sql """drop catalog if exists ${catalogName}""" + sql """ + ${catalogProps} + """ + def result = sql """ + select count(1) from ${catalogName}.${queryTableName}; + """ + println("result: ${result}") + def countValue = result[0][0] + assertTrue(countValue == expectCounts.toInteger()) + sql """drop catalog if exists ${catalogName}""" + } + String suiteName = "test_catalog_with_role" + String awsLakersRoleArn = context.config.otherConfigs.get("awsLakesRoleArn") + String awsLakesRegion = context.config.otherConfigs.get("awsLakesRegion") + String icebergFSOnS3CatalogWarehouse = context.config.otherConfigs.get("icebergFSOnS3CatalogWarehouse") + //String hmsMetastoreHost = context.config.hmsMetastoreHost + //String icebergHmsTableName = context.config.icebergHmsTableName + String icebergGlueTableName = context.config.otherConfigs.get("icebergGlueTableName") + + String icebergFSOnS3TableName = context.config.otherConfigs.get("icebergFSOnS3TableName") + //String hiveHmsTableName = context.config.hiveHmsTableName + String hiveGlueTableName = context.config.otherConfigs.get("hiveGlueTableName") + String expectCounts = context.config.otherConfigs.get("awsLakesRoleArnQueryExpectCounts") + + String awsS3Property = """ + "s3.region" = "${awsLakesRegion}", + "s3.role_arn" = "${awsLakersRoleArn}" + """ + String awsGlueProperties = """ + "glue.region"="${awsLakesRegion}", + "glue.endpoint" = "https://glue.${awsLakesRegion}.amazonaws.com", + "glue.role_arn" = "${awsLakersRoleArn}" + """ + // start test + String catalogName = "${suiteName}_iceberg_hadoop_on_s3" + //1. iceberg hadoop catalog on s + String icebergFSCatalogProps = """ + create catalog if not exists ${catalogName} properties( + "type"="iceberg", + "iceberg.catalog.type"="hadoop", + "warehouse"="${icebergFSOnS3CatalogWarehouse}", + ${awsS3Property} + ); + """ + createCatalogAndQuery(icebergFSCatalogProps, catalogName, icebergFSOnS3TableName, expectCounts) + catalogName = "${suiteName}_iceberg_glue" + String icebergGlueCatalogProps = """ + create catalog if not exists ${catalogName} properties( + "type"="iceberg", + "iceberg.catalog.type"="glue", + ${awsGlueProperties} + ); + """ + createCatalogAndQuery(icebergGlueCatalogProps, catalogName, icebergGlueTableName, expectCounts) + catalogName = "${suiteName}_hive_glue" + String hiveGlueCatalogProps = """ + create catalog if not exists ${catalogName} properties ( + "type"="hms", + "hive.metastore.type" = "glue", + ${awsGlueProperties} + ); + """ + createCatalogAndQuery(hiveGlueCatalogProps, catalogName, hiveGlueTableName, expectCounts) +/*********** HMS ***************/ +// The following HMS-related cases are commented out because +// they require a Docker environment for the Hive Metastore service. + +/* +String catalogName = "${suiteName}iceberg_hive_on_s3" +String icebergHmsCatalogProps = """ + create catalog if not exists ${catalogName} properties ( + "type" = "iceberg", + "iceberg.catalog.type" = "hive", + "hive.metastore.uris" = "thrift://${hmsMetastoreHost}:9083", + ${awsS3Property} + ); +""" +createCatalogAndQuery(icebergHmsCatalogProps, catalogName, icebergHmsTableName, expectCounts) + +String hiveHmsCatalogProps = """ + create catalog if not exists ${catalogName} properties ( + "type" = "hms", + "hive.metastore.uris" = "thrift://${hmsMetastoreHost}:9083", + ${awsS3Property} + ); +""" +createCatalogAndQuery(hiveHmsCatalogProps, catalogName, hiveHmsTableName, expectCounts) +*/ + + +} diff --git a/regression-test/suites/object_storage_iam_p0/test_export_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_export_iam.groovy new file mode 100644 index 00000000000000..046f909b22f2b4 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_export_iam.groovy @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import com.google.common.base.Strings +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_export_iam") { + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def cases = config.authCases.collect { authCase -> + return [ + name: authCase.name, + properties: """ + ${authCase.storageSqlProperties} + """ + ] + } + + def tableName = "test_export_iam" + sql "DROP TABLE IF EXISTS ${tableName} FORCE" + sql """ + CREATE TABLE ${tableName} ( + siteid INT DEFAULT '10', + citycode SMALLINT NOT NULL, + username VARCHAR(32) DEFAULT '', + pv BIGINT SUM DEFAULT '0' + ) + AGGREGATE KEY(siteid, citycode, username) + DISTRIBUTED BY HASH(siteid) BUCKETS 1 + PROPERTIES("replication_num" = "1") + """ + sql """INSERT INTO ${tableName}(siteid, citycode, username, pv) VALUES + (1, 1, "xxx", 1), (2, 2, "yyy", 2), (3, 3, "zzz", 3)""" + + cases.each { testCase -> + def randomStr = UUID.randomUUID().toString().replace("-", "") + def label = "iam_export_${testCase.name}_${randomStr}" + // Keep cleanup inside this run's UUID directory. + def exportPrefix = "${config.prefix}/test_export_iam/${testCase.name}/${randomStr}/" + + logger.info("run ${name} with ${testCase.name}") + sql """ + EXPORT TABLE ${tableName} TO "${config.scheme}://${config.bucket}/${exportPrefix}" + PROPERTIES( + "label" = "${label}", + "format" = "csv", + "column_separator" = ",", + "delete_existing_files" = "true" + ) + WITH S3 ( + ${testCase.properties} + ) + """ + + def maxTryMs = 600000 + def exportedUrl = "" + while (maxTryMs > 0) { + String[][] exportResult = sql "SHOW EXPORT WHERE LABEL = '${label}'" + if (exportResult[0][2] == "FINISHED") { + def json = parseJson(exportResult[0][11]) + assert json instanceof List + assertEquals("1", json.fileNumber[0][0], testCase.name) + exportedUrl = json.url[0][0] + break + } + if (exportResult[0][2] == "CANCELLED") { + assertTrue(false, "Export ${label} cancelled for ${testCase.name}: ${exportResult}") + } + Thread.sleep(5000) + maxTryMs -= 5000 + } + assertTrue(maxTryMs > 0, "Export ${label} timeout for ${testCase.name}") + assertFalse(Strings.isNullOrEmpty(exportedUrl), "Export returned no URL for ${testCase.name}") + + def s3Prefix = "s3://${config.bucket}/" + assertTrue(exportedUrl.startsWith(s3Prefix), "Unexpected export URL: ${exportedUrl}") + def exportedFile = "${config.scheme}://${config.bucket}/" + + "${exportedUrl.substring(s3Prefix.length())}0.csv" + def result = sql """ + SELECT COUNT(*) FROM s3( + "uri" = "${exportedFile}", + ${testCase.properties}, + "format" = "csv", + "column_separator" = ",", + "use_path_style" = "false" + ) + """ + assertEquals(3, result[0][0], testCase.name) + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy new file mode 100644 index 00000000000000..941db827f36625 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy @@ -0,0 +1,78 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_external_catalog_iam") { + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def cases = config.authCases.collect { authCase -> + return [ + name: authCase.name, + warehouse: "${config.scheme}://${config.bucket}/${config.prefix}" + + "/test_external_catalog_iam/${authCase.name}", + properties: """ + ${authCase.storageSqlProperties} + """ + ] + } + + cases.eachWithIndex { testCase, index -> + def randomStr = UUID.randomUUID().toString().replace("-", "") + def catalogName = "iam_iceberg_${index}_${randomStr}" + def databaseName = "iam_db" + def tableName = "iam_table" + + logger.info("run ${name} with ${testCase.name}") + try { + sql """ + CREATE CATALOG ${catalogName} PROPERTIES ( + "type" = "iceberg", + "iceberg.catalog.type" = "hadoop", + "warehouse" = "${testCase.warehouse}/${randomStr}", + ${testCase.properties} + ) + """ + sql "CREATE DATABASE ${catalogName}.${databaseName}" + def databases = sql "SHOW DATABASES FROM ${catalogName}" + assertTrue(databases.any { it[0] == databaseName }, testCase.name) + sql """ + CREATE TABLE ${catalogName}.${databaseName}.${tableName} ( + id INT, + value STRING + ) + PROPERTIES("file_format" = "parquet") + """ + sql "INSERT INTO ${catalogName}.${databaseName}.${tableName} VALUES (1, 'one'), (2, 'two')" + def result = sql "SELECT COUNT(*) FROM ${catalogName}.${databaseName}.${tableName}" + assertEquals(2, result[0][0], testCase.name) + def rows = sql "SELECT id, value FROM ${catalogName}.${databaseName}.${tableName} ORDER BY id" + assertEquals([[1, 'one'], [2, 'two']], rows, testCase.name) + sql "DROP TABLE ${catalogName}.${databaseName}.${tableName}" + sql "DROP DATABASE ${catalogName}.${databaseName}" + // Exercise namespace deletion and recreation through Hadoop FileSystem as well. + sql "CREATE DATABASE ${catalogName}.${databaseName}" + sql "DROP DATABASE ${catalogName}.${databaseName}" + } finally { + sql "DROP CATALOG IF EXISTS ${catalogName}" + } + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_gcp_iam_property_validation.groovy b/regression-test/suites/object_storage_iam_p0/test_gcp_iam_property_validation.groovy new file mode 100644 index 00000000000000..2233a0b881112a --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_gcp_iam_property_validation.groovy @@ -0,0 +1,131 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_gcp_iam_property_validation") { + ["ADC", "", "unknown"].each { providerType -> + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "provider" = "GCP", + "gs.credential_provider_type" = "${providerType}", + "format" = "csv" + ) LIMIT 1 + """ + exception "gs.credential_provider_type" + } + } + + [" ", "invalid-account"].each { account -> + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "provider" = "GCP", + "gs.impersonation_service_account" = "${account}", + "format" = "csv" + ) LIMIT 1 + """ + exception "Invalid GCP service account email" + } + } + + [ + "gs.access_key": "gcp-access-key", + "gs.secret_key": "gcp-secret-key", + "gs.session_token": "gcp-session-token", + "s3.access_key": "access-key", + "s3.secret_key": "secret-key", + "s3.session_token": "session-token", + "s3.role_arn": "arn:aws:iam::123456789012:role/test-role", + "s3.external_id": "external-id", + "s3.credentials_provider_type": "INSTANCE_PROFILE" + ].each { key, value -> + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "provider" = "GCP", + "gs.credential_provider_type" = "DEFAULT", + "${key}" = "${value}", + "format" = "csv" + ) LIMIT 1 + """ + exception "cannot be used together" + } + } + + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "provider" = "S3", + "gs.credential_provider_type" = "DEFAULT", + "format" = "csv" + ) LIMIT 1 + """ + exception "requires provider=GCP" + } + + [ + "gs.access_key": "access-key", + "gs.secret_key": "secret-key", + "gs.session_token": "session-token", + "s3.role_arn": "arn:aws:iam::123456789012:role/test-role" + ].each { key, value -> + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "provider" = "GCP", + "gs.credential_provider_type" = "ANONYMOUS", + "${key}" = "${value}", + "format" = "csv" + ) LIMIT 1 + """ + exception "cannot be used together" + } + } + + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "gs.credential_provider_type" = "ANONYMOUS", + "gs.impersonation_service_account" = "target@project.iam.gserviceaccount.com", + "format" = "csv" + ) LIMIT 1 + """ + exception "ANONYMOUS cannot be used with gs.impersonation_service_account" + } + + // Implicit DEFAULT must enforce the same conflicts as explicit DEFAULT, before any remote I/O. + ["gs.session_token", "s3.role_arn", "s3.credentials_provider_type"].each { key -> + test { + sql """ + SELECT * FROM s3( + "uri" = "gs://validation-only-bucket/file.csv", + "provider" = "GCP", + "${key}" = "conflicting-credential", + "format" = "csv" + ) LIMIT 1 + """ + exception "cannot be used together" + } + } + +} diff --git a/regression-test/suites/object_storage_iam_p0/test_gcp_tvf_anonymous.groovy b/regression-test/suites/object_storage_iam_p0/test_gcp_tvf_anonymous.groovy new file mode 100644 index 00000000000000..459bcb4959245d --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_gcp_tvf_anonymous.groovy @@ -0,0 +1,51 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_gcp_tvf_anonymous") { + def uri = context.config.otherConfigs.get("objectStorageIamGcpAnonymousUri")?.toString()?.trim() + if (!uri) { + logger.info("skip ${name} because objectStorageIamGcpAnonymousUri is not configured") + return + } + assertTrue(uri.startsWith("gs://"), "The anonymous fixture must be a gs:// URI") + def expectedRows = context.config.otherConfigs.get("objectStorageIamGcpAnonymousExpectedRows") + ?.toString()?.trim() + assertTrue(expectedRows != null && expectedRows.isLong() && expectedRows.toLong() > 0, + "Configure a positive objectStorageIamGcpAnonymousExpectedRows") + + // Each query exercises FE object discovery/schema inference and a BE scan of the public file. + // Include the legacy S3-compatible spelling and provider-value case/whitespace normalization. + [ + ["gs.credential_provider_type": "ANONYMOUS"], + ["gs.credential_provider_type": " anonymous "], + ["s3.credentials_provider_type": "ANONYMOUS"] + ].each { authProperties -> + def result = sql """ + SELECT COUNT(*) FROM s3( + "uri" = "${uri}", + "provider" = "GCP", + ${ObjectStorageIamTestUtils.toSqlProperties(authProperties)}, + "format" = "csv", + "compress_type" = "gz", + "column_separator" = "|" + ) + """ + assertEquals(expectedRows.toLong(), result[0][0], authProperties.toString()) + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_gcp_vault_anonymous_validation.groovy b/regression-test/suites/object_storage_iam_p0/test_gcp_vault_anonymous_validation.groovy new file mode 100644 index 00000000000000..f3d699e678d1a1 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_gcp_vault_anonymous_validation.groovy @@ -0,0 +1,42 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_gcp_vault_anonymous_validation") { + if (!isCloudMode() || !enableStoragevault()) { + logger.info("skip ${name} because cloud storage vaults are not enabled") + return + } + + ["gs.credential_provider_type", "s3.credentials_provider_type", "AWS_CREDENTIALS_PROVIDER_TYPE"].each { key -> + def vaultName = "gcp_anonymous_${UUID.randomUUID().toString().replace('-', '')}" + test { + sql """ + CREATE STORAGE VAULT ${vaultName} PROPERTIES ( + "type" = "S3", + "provider" = "GCP", + "s3.endpoint" = "storage.googleapis.com", + "s3.region" = "us-east1", + "s3.bucket" = "validation-only-bucket", + "s3.root.path" = "validation-only-root", + "s3_validity_check" = "false", + "${key}" = " anonymous " + ) + """ + exception "Anonymous GCS authentication is not supported for storage vaults" + } + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy b/regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy new file mode 100644 index 00000000000000..bbef874246c836 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy @@ -0,0 +1,110 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import com.google.common.base.Strings; +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_iceberg_s3tables_catalog_credentials_provider") { + if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { + logger.info("skip ${name} because objectStorageIamProvider is not AWS") + return + } + + List requiredConfigNames = [ + "icebergS3TablesCatalog1Props", + "icebergS3TablesCatalog2Props", + "icebergS3TablesCatalog3Props", + "icebergS3TablesPartitionDbTableName", + "icebergS3TablesNonPartitionDbTableName", + "icebergS3TablesExpectCounts" + ] + if (requiredConfigNames.any { Strings.isNullOrEmpty(context.config.otherConfigs.get(it)) }) { + return + } + + sql """ ADMIN SET FRONTEND CONFIG ("aws_credentials_provider_version"="v2"); """ + + String catalog1Props = context.config.otherConfigs.get("icebergS3TablesCatalog1Props") + String catalog2Props = context.config.otherConfigs.get("icebergS3TablesCatalog2Props") + String catalog3Props = context.config.otherConfigs.get("icebergS3TablesCatalog3Props") + String partitionDbTableName = context.config.otherConfigs.get("icebergS3TablesPartitionDbTableName") + String nonPartitionDbTableName = context.config.otherConfigs.get("icebergS3TablesNonPartitionDbTableName") + String expectCounts = context.config.otherConfigs.get("icebergS3TablesExpectCounts") + String expectedWebIdentityError = "Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set." + + def createCatalogAndQueryTables = { + catalogProps, catalogName, partitionQueryDbTableName, nonPartitionQueryDbTableName, expectedCounts -> + sql """drop catalog if exists ${catalogName}""" + try { + sql """ + create catalog ${catalogName} properties ( + ${catalogProps} + ); + """ + def partitionResult = sql """ + select count(1) from ${catalogName}.${partitionQueryDbTableName}; + """ + def nonPartitionResult = sql """ + select count(1) from ${catalogName}.${nonPartitionQueryDbTableName}; + """ + def partitionCount = partitionResult[0][0] + def nonPartitionCount = nonPartitionResult[0][0] + assertTrue(partitionCount == expectedCounts.toInteger()) + assertTrue(nonPartitionCount == expectedCounts.toInteger()) + assertTrue(partitionCount == nonPartitionCount) + } finally { + sql """drop catalog if exists ${catalogName}""" + } + } + + def createCatalogAndShowDatabasesException = { catalogProps, catalogName, expectedMessage -> + sql """drop catalog if exists ${catalogName}""" + try { + sql """ + create catalog ${catalogName} properties ( + ${catalogProps} + ); + """ + sql """switch ${catalogName};""" + try { + sql """show databases;""" + assertTrue(false) + } catch (Exception e) { + assertTrue(e.getMessage().contains(expectedMessage)) + } + } finally { + sql """drop catalog if exists ${catalogName}""" + } + } + + createCatalogAndQueryTables( + catalog1Props, + "iceberg_s3tables_catalog_1", + partitionDbTableName, + nonPartitionDbTableName, + expectCounts) + createCatalogAndQueryTables( + catalog2Props, + "iceberg_s3tables_catalog_2", + partitionDbTableName, + nonPartitionDbTableName, + expectCounts) + createCatalogAndShowDatabasesException( + catalog3Props, + "iceberg_s3tables_catalog_3", + expectedWebIdentityError) +} diff --git a/regression-test/suites/object_storage_iam_p0/test_load_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_load_iam.groovy new file mode 100644 index 00000000000000..e9b1517305f4db --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_load_iam.groovy @@ -0,0 +1,106 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_load_iam") { + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def cases = config.authCases.collect { authCase -> + return [ + name: authCase.name, + properties: """ + ${authCase.storageSqlProperties} + """ + ] + } + if (config.provider == "AWS") { + cases.add(0, [ + name: "aws_role_legacy_properties", + properties: """ + "AWS_ENDPOINT" = "${config.endpoint}", + "AWS_REGION" = "${config.region}", + "AWS_ROLE_ARN" = "${config.roleArn}", + "AWS_EXTERNAL_ID" = "${config.externalId ?: ''}" + """ + ]) + cases[1].name = "aws_role_s3_properties" + } + + cases.eachWithIndex { testCase, index -> + def randomStr = UUID.randomUUID().toString().replace("-", "") + def loadLabel = "iam_load_${index}_${randomStr}" + def tableName = "test_iam_load_${index}_${randomStr}" + + sql "DROP TABLE IF EXISTS ${tableName} FORCE" + sql """ + CREATE TABLE ${tableName} ( + C_CUSTKEY INTEGER NOT NULL, + C_NAME VARCHAR(25) NOT NULL, + C_ADDRESS VARCHAR(40) NOT NULL, + C_NATIONKEY INTEGER NOT NULL, + C_PHONE CHAR(15) NOT NULL, + C_ACCTBAL DECIMAL(15, 2) NOT NULL, + C_MKTSEGMENT CHAR(10) NOT NULL, + C_COMMENT VARCHAR(117) NOT NULL + ) + DUPLICATE KEY(C_CUSTKEY, C_NAME) + DISTRIBUTED BY HASH(C_CUSTKEY) BUCKETS 1 + PROPERTIES("replication_num" = "1") + """ + + logger.info("run ${name} with ${testCase.name}") + sql """ + LOAD LABEL ${loadLabel} ( + DATA INFILE("${config.scheme}://${config.bucket}/${config.dataPath}") + INTO TABLE ${tableName} + COLUMNS TERMINATED BY "|" + (c_custkey, c_name, c_address, c_nationkey, c_phone, c_acctbal, c_mktsegment, c_comment) + ) + WITH S3 ( + ${testCase.properties}, + "compress_type" = "GZ" + ) + PROPERTIES( + "timeout" = "600", + "exec_mem_limit" = "8589934592" + ) + """ + + def maxTryMs = 600000 + while (maxTryMs > 0) { + String[][] loadResult = sql """SHOW LOAD WHERE LABEL = "${loadLabel}" + ORDER BY CREATETIME DESC LIMIT 1""" + if (loadResult[0][2] == "FINISHED") { + break + } + if (loadResult[0][2] == "CANCELLED") { + assertTrue(false, "Load ${loadLabel} cancelled for ${testCase.name}: ${loadResult}") + } + Thread.sleep(5000) + maxTryMs -= 5000 + } + assertTrue(maxTryMs > 0, "Load ${loadLabel} timeout for ${testCase.name}") + + def result = sql "SELECT COUNT(*) FROM ${tableName}" + assertEquals(1500, result[0][0], testCase.name) + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_resource_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_resource_iam.groovy new file mode 100644 index 00000000000000..7433179e902fbe --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_resource_iam.groovy @@ -0,0 +1,134 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import groovy.json.JsonSlurper +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_resource_iam") { + if (isCloudMode()) { + logger.info("skip ${name} case, because cloud mode not support") + return + } + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def randomStr = UUID.randomUUID().toString().replace("-", "") + def testObjects = [] + config.authCases.eachWithIndex { authCase, index -> + def tableName = "test_resource_iam_${index}" + def resourceName = "resource_${authCase.name}_${randomStr}" + def policyName = "policy_${authCase.name}_${randomStr}" + + logger.info("run ${name} with ${authCase.name}") + sql """ + CREATE RESOURCE IF NOT EXISTS "${resourceName}" + PROPERTIES( + "type" = "s3", + ${authCase.storageSqlProperties}, + "s3.bucket" = "${config.bucket}", + "s3.root.path" = "${config.prefix}/test_resource_iam/${authCase.name}/${randomStr}", + "s3.connection.maximum" = "50", + "s3.connection.request.timeout" = "3000", + "s3.connection.timeout" = "1000", + "s3_validity_check" = "true" + ); + """ + + sql """ + CREATE STORAGE POLICY IF NOT EXISTS ${policyName} + PROPERTIES( + "storage_resource" = "${resourceName}", + "cooldown_ttl" = "1" + ) + """ + + sql "DROP TABLE IF EXISTS ${tableName} FORCE" + sql """ + CREATE TABLE ${tableName} + ( + siteid INT DEFAULT '10', + citycode SMALLINT NOT NULL, + username VARCHAR(32) DEFAULT '', + pv BIGINT SUM DEFAULT '0' + ) + AGGREGATE KEY(siteid, citycode, username) + DISTRIBUTED BY HASH(siteid) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1", + "storage_policy" = "${policyName}" + ) + """ + + sql """insert into ${tableName}(siteid, citycode, username, pv) values (1, 1, "xxx", 1), + (2, 2, "yyy", 2), + (3, 3, "zzz", 3) + """ + testObjects.add([name: authCase.name, tableName: tableName]) + } + + // data_sizes is one arrayList, t is tablet + def fetchDataSize = {List data_sizes, Map t -> + def tabletId = t.TabletId + def meta_url = t.MetaUrl + def clos = { respCode, body -> + logger.info("test ttl expired resp Code {}", "${respCode}".toString()) + assertEquals("${respCode}".toString(), "200") + String out = "${body}".toString() + def obj = new JsonSlurper().parseText(out) + data_sizes[0] = obj.local_data_size + data_sizes[1] = obj.remote_data_size + } + meta_url = meta_url.replace("header", "data_size") + + def i = meta_url.indexOf("/api") + def endPoint = meta_url.substring(0, i) + def metaUri = meta_url.substring(i) + logger.info("test fetchBeHttp, endpoint:${endPoint}, metaUri:${metaUri}") + i = endPoint.lastIndexOf('/') + endPoint = endPoint.substring(i + 1) + + httpTest { + endpoint {endPoint} + uri metaUri + op "get" + check clos + } + } + + sleep(60000) + + testObjects.each { testObject -> + List sizes = [-1, -1] + def tablets = sql_return_maparray "SHOW TABLETS FROM ${testObject.tableName}" + log.info("test tablets not empty for ${testObject.name}: ${tablets}") + fetchDataSize(sizes, tablets[0]) + def retry = 100 + while (sizes[1] == 0 && retry-- > 0) { + log.info("test remote size is zero for ${testObject.name}, sleep 10s") + sleep(10000) + tablets = sql_return_maparray "SHOW TABLETS FROM ${testObject.tableName}" + fetchDataSize(sizes, tablets[0]) + } + assertTrue(sizes[1] != 0, + "remote size is still zero for ${testObject.name}, maybe some error occurred") + assertTrue(tablets.size() > 0) + log.info("test remote size not zero for ${testObject.name}") + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_select_into_outfile_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_select_into_outfile_iam.groovy new file mode 100644 index 00000000000000..e71067ab2879b9 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_select_into_outfile_iam.groovy @@ -0,0 +1,80 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_select_into_outfile_iam") { + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def randomStr = UUID.randomUUID().toString().replace("-", "") + def tableName = "test_select_into_outfile_iam" + + sql """ drop table if exists ${tableName} force;""" + sql """ + CREATE TABLE ${tableName} + ( + siteid INT DEFAULT '10', + citycode SMALLINT NOT NULL, + username VARCHAR(32) DEFAULT '', + pv BIGINT SUM DEFAULT '0' + ) + AGGREGATE KEY(siteid, citycode, username) + DISTRIBUTED BY HASH(siteid) BUCKETS 1 + PROPERTIES ( + "replication_num" = "1" + ) + """ + + sql """insert into ${tableName}(siteid, citycode, username, pv) values (1, 1, "xxx", 1), + (2, 2, "yyy", 2), + (3, 3, "zzz", 3) + """ + sql """sync;""" + + def expectedRows = sql """ + SELECT CAST(siteid AS STRING), CAST(citycode AS STRING), username, CAST(pv AS STRING) + FROM ${tableName} ORDER BY siteid + """ + + config.authCases.each { authCase -> + logger.info("run ${name} with ${authCase.name}") + def outfilePath = "${config.scheme}://${config.bucket}/${config.prefix}/" + + "test_select_into_outfile_iam/${authCase.name}/${randomStr}" + sql """ + SELECT * FROM ${tableName} + INTO OUTFILE "${outfilePath}" + FORMAT AS CSV + PROPERTIES( + "column_separator" = ",", + ${authCase.storageSqlProperties} + ); + """ + def exportedRows = sql """ + SELECT c1, c2, c3, c4 FROM s3( + "uri" = "${outfilePath}*.csv", + ${authCase.storageSqlProperties}, + "format" = "csv", + "column_separator" = "," + ) ORDER BY CAST(c1 AS INT) + """ + assertEquals(expectedRows, exportedRows, authCase.name) + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_storage_vault_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_storage_vault_iam.groovy new file mode 100644 index 00000000000000..9f89873ef2cd37 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_storage_vault_iam.groovy @@ -0,0 +1,80 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_storage_vault_iam") { + if (!isCloudMode()) { + logger.info("skip ${name} because it requires cloud mode") + return + } + if (!enableStoragevault()) { + logger.info("skip ${name} because storage vault is disabled") + return + } + + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + def cases = config.authCases.collect { authCase -> + return [ + name: authCase.name, + properties: """ + ${authCase.storageSqlProperties}, + "s3.bucket" = "${config.bucket}", + "s3.external_endpoint" = "", + "use_path_style" = "false" + """ + ] + } + + cases.eachWithIndex { testCase, index -> + def randomStr = UUID.randomUUID().toString().replace("-", "") + def vaultName = "iam_vault_${index}_${randomStr}" + def tableName = "test_iam_vault_${index}_${randomStr}" + + logger.info("run ${name} with ${testCase.name}") + sql """ + CREATE STORAGE VAULT ${vaultName} + PROPERTIES ( + "type" = "S3", + "s3.root.path" = "${config.prefix}/test_storage_vault_iam/${testCase.name}/${randomStr}", + ${testCase.properties} + ) + """ + sql """ + CREATE TABLE ${tableName} ( + id INT NOT NULL, + value INT NOT NULL + ) + DUPLICATE KEY(id, value) + DISTRIBUTED BY HASH(id) BUCKETS 1 + PROPERTIES( + "replication_num" = "1", + "storage_vault_name" = "${vaultName}" + ) + """ + sql "INSERT INTO ${tableName} VALUES (1, 1)" + sql "SYNC" + def result = sql "SELECT * FROM ${tableName}" + assertEquals(1, result.size(), testCase.name) + assertEquals(1, result[0][0], testCase.name) + } +} diff --git a/regression-test/suites/object_storage_iam_p0/test_tvf_anonymous.groovy b/regression-test/suites/object_storage_iam_p0/test_tvf_anonymous.groovy new file mode 100644 index 00000000000000..5cc3fd392f0957 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_tvf_anonymous.groovy @@ -0,0 +1,77 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + + +import com.google.common.base.Strings; +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_tvf_anonymous") { + if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { + logger.info("skip ${name} because objectStorageIamProvider is not AWS") + return + } + if (Strings.isNullOrEmpty(context.config.otherConfigs.get("enableTestTvfAnonymous"))) { + return + } + + + def region = context.config.otherConfigs.get("anymousS3Region") + def uri = context.config.otherConfigs.get("anymousS3Uri") + def expectDataCount = context.config.otherConfigs.get("anymousS3ExpectDataCount"); + //aws_credentials_provider_version + // sql """ ADMIN SET FRONTEND CONFIG ("aws_credentials_provider_version"="v1"); """ + + def result = sql """ + SELECT count(1) FROM S3 ( + "uri"="${uri}", + "format" = "csv", + "s3.region" = "${region}", + "s3.credentials_provider_type"="ANONYMOUS", + "s3.endpoint" = "https://s3.${region}.amazonaws.com", + "column_separator" = "," ); + """ + + def countValue = result[0][0] + assertTrue(countValue == expectDataCount.toInteger()) + sql """ ADMIN SET FRONTEND CONFIG ("aws_credentials_provider_version"="v2"); """ + + result = sql """ + SELECT count(1) FROM S3 ( + "uri"="${uri}", + "format" = "csv", + "s3.region" = "${region}", + "s3.credentials_provider_type"="ANONYMOUS", + "s3.endpoint" = "https://s3.${region}.amazonaws.com", + "column_separator" = "," ); + """ + + countValue = result[0][0] + assertTrue(countValue == expectDataCount.toInteger()) + + result = sql """ + SELECT count(1) FROM S3 ( + "uri"="${uri}", + "format" = "csv", + "s3.region" = "${region}", + "s3.endpoint" = "https://s3.${region}.amazonaws.com", + "s3.credentials_provider_type"="ANONYMOUS", + "column_separator" = "," ); + """ + + countValue = result[0][0] + assertTrue(countValue == expectDataCount.toInteger()) +} diff --git a/regression-test/suites/object_storage_iam_p0/test_tvf_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_tvf_iam.groovy new file mode 100644 index 00000000000000..1a28289e55c035 --- /dev/null +++ b/regression-test/suites/object_storage_iam_p0/test_tvf_iam.groovy @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +import org.apache.doris.regression.util.ObjectStorageIamTestUtils + +suite("test_tvf_iam") { + def config = ObjectStorageIamTestUtils.getConfig(context.config.otherConfigs) + if (config == null) { + logger.info("skip ${name} because objectStorageIamProvider is not configured") + return + } + + config.authCases.each { authCase -> + logger.info("run ${name} with ${authCase.name}") + def result = sql """ + SELECT COUNT(*) FROM s3( + "uri" = "${config.scheme}://${config.bucket}/${config.dataPath}", + ${authCase.storageSqlProperties}, + "format" = "csv", + "compress_type" = "gz", + "column_separator" = "|", + "use_path_style" = "false" + ) + """ + assertEquals(1500, result[0][0], authCase.name) + } + + if (config.provider == "GCP" && config.authCases.any { it.name == "gcp_default_omitted" }) { + // The preceding read succeeded with omitted properties. The same private object must not + // become readable with ANONYMOUS, even on a VM with a usable attached service account. + test { + sql """ + SELECT COUNT(*) FROM s3( + "uri" = "gs://${config.bucket}/${config.dataPath}", + "provider" = "GCP", + "gs.endpoint" = "${config.endpoint}", + "gs.credential_provider_type" = "ANONYMOUS", + "format" = "csv", + "compress_type" = "gz", + "column_separator" = "|" + ) + """ + check { result, exception, startTime, endTime -> + assertTrue(exception != null, "Anonymous access unexpectedly read the private GCS fixture") + def message = exception.toString().toLowerCase() + assertTrue(message.contains("403") || message.contains("access denied") + || message.contains("accessdenied") || message.contains("forbidden") + || message.contains("permission denied"), message) + } + } + } +} From 6ddc93219c37b78b9718a1531eed9169682ea3f7 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 28 Sep 2026 14:46:14 +0800 Subject: [PATCH 02/47] [fix](storage) Honor native GCP Hadoop settings and reject token failures ### What problem does this PR solve? Related PR: #68540 Problem Summary: Native GCP Hadoop access ignored the configured endpoint and HadoopCatalog selected S3A for s3/s3a warehouse aliases. Propagate the endpoint to the Google connector and normalize native HadoopCatalog warehouses to gs. The C++ client also sent anonymous object requests when token acquisition failed. Acquire bearer tokens before each object-storage SDK operation and return a non-retryable authentication error on missing or empty tokens. Add endpoint, warehouse alias, and authentication-failure coverage. ### Release note Native GCP catalogs honor custom endpoints and compatible warehouse schemes. Missing GCP tokens now fail locally instead of issuing anonymous requests. ### Check List (For Author) - Test: GCS module and upstream unit tests passed (37 GCS tests); FE Checkstyle, clang-format 16 on changed C++ files, build hygiene, and git diff --check passed. Iceberg tests stopped at the Thrift 0.16/0.24 mismatch; BE test setup stopped at the JDK 17 requirement. Further compilation and tests skipped at user request. Live GCP regression tests were not run. - Behavior changed: Yes, native Hadoop endpoint/scheme routing and token failure handling. - Does this need documentation: No --- be/test/io/s3_client_factory_test.cpp | 58 ++++++- .../cpp/obj-client/auth/gcp/gcp_s3_client.cpp | 153 ++++++++++++++++-- .../cpp/obj-client/auth/gcp/gcp_s3_client.h | 47 +++++- .../iceberg/IcebergCatalogFactory.java | 11 ++ .../iceberg/IcebergCatalogFactoryTest.java | 19 +++ .../gcs/GcsFileSystemProperties.java | 5 + .../gcs/GcsFileSystemPropertiesTest.java | 12 ++ .../test_external_catalog_iam.groovy | 21 +-- 8 files changed, 301 insertions(+), 25 deletions(-) diff --git a/be/test/io/s3_client_factory_test.cpp b/be/test/io/s3_client_factory_test.cpp index c996cd4f105bb2..a1cfde54e77e8f 100644 --- a/be/test/io/s3_client_factory_test.cpp +++ b/be/test/io/s3_client_factory_test.cpp @@ -23,8 +23,30 @@ #include #include #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +#include +#include #include +#include +#include +#include +#include +#include +#include #include #include @@ -33,6 +55,7 @@ #include #include #include +#include #include #include #include @@ -154,9 +177,12 @@ class FixedTokenGcpS3Client final : public GcpS3Client { Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true) {} using GcpS3Client::BuildHttpRequest; + using GcpS3Client::authorize_request; + + std::optional token = "test-gcp-token"; protected: - std::optional fetch_token() const override { return "test-gcp-token"; } + std::optional fetch_token() const override { return token; } }; class SyncPointProcessingGuard { @@ -381,6 +407,7 @@ TEST_F(S3ClientFactoryTest, GcpBearerHeaderSurvivesAnonymousAwsSigning) { Aws::Http::HttpMethod::HTTP_GET, Aws::Utils::Stream::DefaultResponseStreamFactoryMethod); + ASSERT_FALSE(client.authorize_request(request).has_value()); client.BuildHttpRequest(request, http_request); EXPECT_EQ(http_request->GetHeaderValue("Authorization"), "Bearer test-gcp-token"); @@ -393,6 +420,35 @@ TEST_F(S3ClientFactoryTest, GcpBearerHeaderSurvivesAnonymousAwsSigning) { EXPECT_EQ(http_request->GetHeaderValue("Authorization"), "Bearer test-gcp-token"); } +TEST_F(S3ClientFactoryTest, GcpTokenFailureRejectsEveryObjectStorageOperation) { + FixedTokenGcpS3Client client; + for (const auto& token : {std::optional {}, std::optional {""}}) { + client.token = token; + auto check = [](const auto& outcome) { + ASSERT_FALSE(outcome.IsSuccess()); + EXPECT_EQ(outcome.GetError().GetExceptionName(), "GcpAuthenticationError"); + EXPECT_EQ(outcome.GetError().GetResponseCode(), + Aws::Http::HttpResponseCode::UNAUTHORIZED); + EXPECT_FALSE(outcome.GetError().ShouldRetry()); + EXPECT_EQ(doris::s3fs_error(outcome.GetError(), "GCS request failed").code, + ObjStorageStatus::PERMISSION_DENIED); + }; + check(client.AbortMultipartUpload(Aws::S3::Model::AbortMultipartUploadRequest {})); + check(client.CompleteMultipartUpload(Aws::S3::Model::CompleteMultipartUploadRequest {})); + check(client.CreateMultipartUpload(Aws::S3::Model::CreateMultipartUploadRequest {})); + check(client.DeleteObject(Aws::S3::Model::DeleteObjectRequest {})); + check(client.DeleteObjects(Aws::S3::Model::DeleteObjectsRequest {})); + check(client.GetBucketLifecycleConfiguration( + Aws::S3::Model::GetBucketLifecycleConfigurationRequest {})); + check(client.GetBucketVersioning(Aws::S3::Model::GetBucketVersioningRequest {})); + check(client.GetObject(Aws::S3::Model::GetObjectRequest {})); + check(client.HeadObject(Aws::S3::Model::HeadObjectRequest {})); + check(client.ListObjectsV2(Aws::S3::Model::ListObjectsV2Request {})); + check(client.PutObject(Aws::S3::Model::PutObjectRequest {})); + check(client.UploadPart(Aws::S3::Model::UploadPartRequest {})); + } +} + TEST_F(S3ClientFactoryTest, UnauthorizedObjectStoreResponseIsPermissionDenied) { Aws::S3::S3Error error; error.SetResponseCode(Aws::Http::HttpResponseCode::UNAUTHORIZED); diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp index 0ad7ba19d2ce26..da2a0fd9b7e1d7 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp @@ -18,6 +18,30 @@ #include "cpp/obj-client/auth/gcp/gcp_s3_client.h" #include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include @@ -31,17 +55,128 @@ GcpS3Client::GcpS3Client(const GcpCredentialConfig& credential, std::string ca_c std::move(aws_config), payload_signing_policy, use_virtual_addressing), _token_provider(credential, ca_cert_path) {} -void GcpS3Client::BuildHttpRequest( - const Aws::AmazonWebServiceRequest& request, - const std::shared_ptr& http_request) const { - Aws::S3::S3Client::BuildHttpRequest(request, http_request); +std::optional GcpS3Client::authorize_request( + Aws::AmazonWebServiceRequest& request) const { auto token = fetch_token(); - if (!token.has_value()) { - // GCS will return a useful 401 response. The token provider has already - // logged the underlying credential or metadata error. - return; + if (!token.has_value() || token->empty()) { + // Return through the SDK outcome path before any object-storage HTTP request. + // The token provider has already logged the underlying credential error. + Aws::S3::S3Error error(Aws::S3::S3Errors::ACCESS_DENIED, "GcpAuthenticationError", + "Failed to obtain GCP access token", false); + error.SetResponseCode(Aws::Http::HttpResponseCode::UNAUTHORIZED); + return error; } - http_request->SetHeaderValue("Authorization", Aws::String("Bearer ") + token->c_str()); + request.SetAdditionalCustomHeaderValue("Authorization", + Aws::String("Bearer ") + token->c_str()); + return std::nullopt; +} + +Aws::S3::Model::AbortMultipartUploadOutcome GcpS3Client::AbortMultipartUpload( + const Aws::S3::Model::AbortMultipartUploadRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::AbortMultipartUpload(authorized_request); +} + +Aws::S3::Model::CompleteMultipartUploadOutcome GcpS3Client::CompleteMultipartUpload( + const Aws::S3::Model::CompleteMultipartUploadRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::CompleteMultipartUpload(authorized_request); +} + +Aws::S3::Model::CreateMultipartUploadOutcome GcpS3Client::CreateMultipartUpload( + const Aws::S3::Model::CreateMultipartUploadRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::CreateMultipartUpload(authorized_request); +} + +Aws::S3::Model::DeleteObjectOutcome GcpS3Client::DeleteObject( + const Aws::S3::Model::DeleteObjectRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::DeleteObject(authorized_request); +} + +Aws::S3::Model::DeleteObjectsOutcome GcpS3Client::DeleteObjects( + const Aws::S3::Model::DeleteObjectsRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::DeleteObjects(authorized_request); +} + +Aws::S3::Model::GetBucketLifecycleConfigurationOutcome GcpS3Client::GetBucketLifecycleConfiguration( + const Aws::S3::Model::GetBucketLifecycleConfigurationRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::GetBucketLifecycleConfiguration(authorized_request); +} + +Aws::S3::Model::GetBucketVersioningOutcome GcpS3Client::GetBucketVersioning( + const Aws::S3::Model::GetBucketVersioningRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::GetBucketVersioning(authorized_request); +} + +Aws::S3::Model::GetObjectOutcome GcpS3Client::GetObject( + const Aws::S3::Model::GetObjectRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::GetObject(authorized_request); +} + +Aws::S3::Model::HeadObjectOutcome GcpS3Client::HeadObject( + const Aws::S3::Model::HeadObjectRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::HeadObject(authorized_request); +} + +Aws::S3::Model::ListObjectsV2Outcome GcpS3Client::ListObjectsV2( + const Aws::S3::Model::ListObjectsV2Request& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::ListObjectsV2(authorized_request); +} + +Aws::S3::Model::PutObjectOutcome GcpS3Client::PutObject( + const Aws::S3::Model::PutObjectRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::PutObject(authorized_request); +} + +Aws::S3::Model::UploadPartOutcome GcpS3Client::UploadPart( + const Aws::S3::Model::UploadPartRequest& request) const { + auto authorized_request = request; + if (auto error = authorize_request(authorized_request)) { + return std::move(*error); + } + return Aws::S3::S3Client::UploadPart(authorized_request); } std::optional GcpS3Client::fetch_token() const { diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.h b/common/cpp/obj-client/auth/gcp/gcp_s3_client.h index 4e15f3255b3c78..8b301dfc483e63 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_s3_client.h +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.h @@ -17,11 +17,11 @@ #pragma once +#include #include -#include #include +#include -#include #include #include @@ -32,7 +32,8 @@ namespace doris { // GCS accepts OAuth2 bearer tokens on its S3-compatible XML API. The client uses // anonymous AWS credentials so AWSAuthV4Signer returns without adding SigV4, and -// BuildHttpRequest injects the bearer header immediately before that signing step. +// each object-storage operation obtains its bearer token before entering the SDK. +// Keep these overrides in sync with the operations used by S3ObjStorageClient. class GcpS3Client : public Aws::S3::S3Client { public: GcpS3Client(const GcpCredentialConfig& credential, std::string ca_cert_path, @@ -40,10 +41,44 @@ class GcpS3Client : public Aws::S3::S3Client { Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy payload_signing_policy, bool use_virtual_addressing); + Aws::S3::Model::AbortMultipartUploadOutcome AbortMultipartUpload( + const Aws::S3::Model::AbortMultipartUploadRequest& request) const override; + + Aws::S3::Model::CompleteMultipartUploadOutcome CompleteMultipartUpload( + const Aws::S3::Model::CompleteMultipartUploadRequest& request) const override; + + Aws::S3::Model::CreateMultipartUploadOutcome CreateMultipartUpload( + const Aws::S3::Model::CreateMultipartUploadRequest& request) const override; + + Aws::S3::Model::DeleteObjectOutcome DeleteObject( + const Aws::S3::Model::DeleteObjectRequest& request) const override; + + Aws::S3::Model::DeleteObjectsOutcome DeleteObjects( + const Aws::S3::Model::DeleteObjectsRequest& request) const override; + + Aws::S3::Model::GetBucketLifecycleConfigurationOutcome GetBucketLifecycleConfiguration( + const Aws::S3::Model::GetBucketLifecycleConfigurationRequest& request) const override; + + Aws::S3::Model::GetBucketVersioningOutcome GetBucketVersioning( + const Aws::S3::Model::GetBucketVersioningRequest& request) const override; + + Aws::S3::Model::GetObjectOutcome GetObject( + const Aws::S3::Model::GetObjectRequest& request) const override; + + Aws::S3::Model::HeadObjectOutcome HeadObject( + const Aws::S3::Model::HeadObjectRequest& request) const override; + + Aws::S3::Model::ListObjectsV2Outcome ListObjectsV2( + const Aws::S3::Model::ListObjectsV2Request& request) const override; + + Aws::S3::Model::PutObjectOutcome PutObject( + const Aws::S3::Model::PutObjectRequest& request) const override; + + Aws::S3::Model::UploadPartOutcome UploadPart( + const Aws::S3::Model::UploadPartRequest& request) const override; + protected: - void BuildHttpRequest( - const Aws::AmazonWebServiceRequest& request, - const std::shared_ptr& http_request) const override; + std::optional authorize_request(Aws::AmazonWebServiceRequest& request) const; // Test seam for asserting the outgoing header without relying on ADC or metadata. virtual std::optional fetch_token() const; diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index 49113697c77d87..662e9a287a3e2d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -429,6 +429,17 @@ public static Map buildCatalogProperties(IcebergCatalogPropertie "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider"); } auth.flatMap(GcsAuth::getNativeCredential).ifPresent(credential -> { + // HadoopCatalog resolves its namespace filesystem independently of S3FileIO. + // Native GCS credentials configure fs.gs.*, so compatibility warehouse schemes + // must select that same filesystem before HadoopCatalog initializes. + String warehouse = opts.get(CatalogProperties.WAREHOUSE_LOCATION); + if (IcebergCatalogProperties.TYPE_HADOOP.equals(flavor) && warehouse != null) { + if (warehouse.regionMatches(true, 0, "s3://", 0, 5)) { + opts.put(CatalogProperties.WAREHOUSE_LOCATION, "gs://" + warehouse.substring(5)); + } else if (warehouse.regionMatches(true, 0, "s3a://", 0, 6)) { + opts.put(CatalogProperties.WAREHOUSE_LOCATION, "gs://" + warehouse.substring(6)); + } + } putS3FileIODialect(opts, storage); opts.put("provider", "GCP"); opts.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index 100305cdc6a400..db6435d673783c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -26,6 +26,8 @@ import org.apache.doris.filesystem.properties.StorageProperties; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.iceberg.aws.AwsClientProperties; import org.junit.jupiter.api.Assertions; @@ -1071,6 +1073,23 @@ void nativeGcpCatalogUsesBearerFileIoAndNativeHadoopConfiguration() { } } + @Test + void nativeGcpWarehouseAliasesSelectGoogleHadoopFileSystem() throws Exception { + for (String scheme : List.of("gs", "s3", "s3a", "S3A")) { + Map raw = Map.of("provider", "GCP", "warehouse", scheme + "://bucket/warehouse", + "iceberg.catalog.type", "hadoop", "gs.credential_provider_type", "COMPUTE_ENGINE"); + GcsFileSystemProperties storage = GcsFileSystemProperties.of(raw); + Map options = IcebergCatalogFactory.buildCatalogProperties( + IcebergCatalogProperties.of(raw), Optional.of(storage)); + String warehouse = options.get("warehouse"); + Assertions.assertEquals("gs://bucket/warehouse", warehouse); + Configuration conf = IcebergCatalogFactory.buildHadoopConfiguration(raw, + storage.toHadoopConfigurationMap()); + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + FileSystem.getFileSystemClass(new Path(warehouse).toUri().getScheme(), conf).getName()); + } + } + @Test void anonymousGcpCatalogUsesUnsignedFileIo() { Map raw = Map.of("provider", "GCP", "warehouse", "gs://bucket/warehouse", diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index b2b7e0f9bbf209..b9e4d5d45e0f38 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -191,6 +191,11 @@ public Map toHadoopConfigurationMap() { GcpCredential credential = auth.getNativeCredential().get(); Map cfg = new HashMap<>(); cfg.put("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + String storageRoot = getEndpoint(); + if (!storageRoot.contains("://")) { + storageRoot = "https://" + storageRoot; + } + cfg.put("fs.gs.storage.root.url", storageRoot.endsWith("/") ? storageRoot : storageRoot + "/"); cfg.put("fs.gs.auth.type", credential.getCredentialProviderType() == GcpCredentialProviderType.DEFAULT ? "APPLICATION_DEFAULT" : "COMPUTE_ENGINE"); cfg.put("fs.gs.auth.impersonation.service.account", credential.getImpersonationServiceAccount()); diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index ff3f78006d878b..684358a1d57305 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -114,6 +114,18 @@ void toHadoopConfigurationMap_usesNativeAdcWithoutStaticCredentials() { Assertions.assertFalse(cfg.containsKey("fs.s3a.aws.credentials.provider")); } + @Test + void nativeHadoopUsesConfiguredEndpoint() { + for (String endpoint : new String[] {"private-gcs.example.test", "https://private-gcs.example.test", + "https://private-gcs.example.test/", "http://localhost:4443"}) { + GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( + "gs.endpoint", endpoint, "gs.credential_provider_type", "COMPUTE_ENGINE")); + String expected = endpoint.contains("://") ? endpoint : "https://" + endpoint; + Assertions.assertEquals(expected.endsWith("/") ? expected : expected + "/", + props.toHadoopConfigurationMap().get("fs.gs.storage.root.url")); + } + } + @Test void toHadoopConfigurationMap_simpleProviderWithStaticCredentials() { Map cfg = GcsFileSystemProperties.of(Map.of( diff --git a/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy b/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy index 941db827f36625..4d26375c0b19e3 100644 --- a/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy +++ b/regression-test/suites/object_storage_iam_p0/test_external_catalog_iam.groovy @@ -24,15 +24,18 @@ suite("test_external_catalog_iam") { return } - def cases = config.authCases.collect { authCase -> - return [ - name: authCase.name, - warehouse: "${config.scheme}://${config.bucket}/${config.prefix}" + - "/test_external_catalog_iam/${authCase.name}", - properties: """ - ${authCase.storageSqlProperties} - """ - ] + def schemes = config.provider == "GCP" ? ["gs", "s3", "s3a"] : [config.scheme] + def cases = config.authCases.collectMany { authCase -> + schemes.collect { scheme -> + return [ + name: "${authCase.name}_${scheme}", + warehouse: "${scheme}://${config.bucket}/${config.prefix}" + + "/test_external_catalog_iam/${authCase.name}_${scheme}", + properties: """ + ${authCase.storageSqlProperties} + """ + ] + } } cases.eachWithIndex { testCase, index -> From 75610a485b4a066a1d63e0c8c3f9182095805588 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 28 Sep 2026 15:20:34 +0800 Subject: [PATCH 03/47] [fix](fe) Let resource ALTER override persisted GCS aliases ### What problem does this PR solve? Related PR: #68540 Problem Summary: Resources saved before GCS alias normalization can retain old gs.* properties. Normalizing only after merging an ALTER lets an old alias override a new s3.* value, silently ignoring credential rotation or restoring a session token that the user cleared. Normalize stored properties and the patch separately before merging, so the patch takes precedence while preserving gs.* priority within the same input map. Update the regression unit test to expect the new access/secret keys and the cleared session token. ### Release note GCP resource ALTER updates now override persisted gs.* aliases correctly. ### Check List (For Author) - Test: FE Core Checkstyle and git diff --check passed. Updated unit coverage for credential replacement, token clearing, unchanged properties, and same-input alias precedence. Compilation and tests skipped at user request. - Behavior changed: Yes, explicit resource updates override old alias values. - Does this need documentation: No --- .../java/org/apache/doris/catalog/S3Resource.java | 13 +++++-------- .../org/apache/doris/catalog/S3ResourceTest.java | 15 ++++++++++----- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index 8e0ddea6ef2c19..02d9049cc6ee97 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -269,7 +269,12 @@ public synchronized void modifyProperties(Map newProperties) thr this.properties.get("provider")); // Preserve AWS_* ALTER compatibility before merging with stored canonical properties. S3ResourceCompat.convertToStdProperties(properties); + // Resolve aliases separately so this ALTER wins over persisted values regardless + // of their spelling. Within each map, nonblank gs.* values still take precedence. + normalizeProperties(properties, provider); Map effectiveProperties = new HashMap<>(this.properties); + normalizeProperties(effectiveProperties, provider); + S3ResourceCompat.convertToStdProperties(effectiveProperties); for (Map.Entry update : properties.entrySet()) { // Match persistence: empty updates are ignored, except when clearing a session token. replaceIfEffectiveValue(effectiveProperties, update.getKey(), update.getValue()); @@ -278,14 +283,6 @@ public synchronized void modifyProperties(Map newProperties) thr effectiveProperties.put(update.getKey(), update.getValue()); } } - // Resolve aliases only after merging: nonblank gs.* values, including stored aliases, - // take precedence over s3.* values in the final configuration. - normalizeProperties(effectiveProperties, provider); - S3ResourceCompat.convertToStdProperties(effectiveProperties); - - // Normalize the update keys for modification restrictions and credential-mode changes. - // Their values must not overwrite the configuration resolved above. - normalizeProperties(properties, provider); if (references.containsValue(ReferenceType.POLICY)) { // can't change, because remote fs use it info to find data. List cantChangeProperties = Arrays.asList(S3ResourceCompat.ENDPOINT, S3ResourceCompat.REGION, diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index e9390b9371ea48..1e83f7cf712d59 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -688,7 +688,7 @@ public void testGsCredentialAliasesCannotBypassNativeAuthValidation() throws Exc } @Test - public void testGsAliasesTakePrecedenceAfterMergingAlter() throws Exception { + public void testAlterOverridesStoredGsAliases() throws Exception { Map properties = new HashMap<>(s3Properties); properties.put("provider", "GCP"); S3Resource resource = new S3Resource("gcp_resource"); @@ -697,25 +697,30 @@ public void testGsAliasesTakePrecedenceAfterMergingAlter() throws Exception { Field field = S3Resource.class.getDeclaredField("properties"); field.setAccessible(true); Map legacy = resource.getCopiedProperties(); + legacy.put("gs.access_key", "old-access"); legacy.put("gs.secret_key", "old-secret"); legacy.put("gs.session_token", "old-token"); legacy.put("gs.connection.timeout", "789"); field.set(resource, legacy); - resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.SECRET_KEY, "new-secret", + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.ACCESS_KEY, "new-access", + S3ResourceCompat.SECRET_KEY, "new-secret", S3ResourceCompat.SESSION_TOKEN, "")); Map normalized = resource.getCopiedProperties(); S3CompatibleFileSystemProperties connector = (S3CompatibleFileSystemProperties) StorageAdapter.of(normalized).getSpiProperties(); - Assertions.assertEquals("old-secret", connector.getSecretKey()); - Assertions.assertEquals("old-token", connector.getSessionToken()); + Assertions.assertEquals("new-access", connector.getAccessKey()); + Assertions.assertEquals("new-secret", connector.getSecretKey()); + Assertions.assertEquals("", connector.getSessionToken()); + Assertions.assertEquals(connector.getAccessKey(), S3ThriftAdapter.getS3TStorageParam(normalized).getAk()); Assertions.assertEquals(connector.getSecretKey(), S3ThriftAdapter.getS3TStorageParam(normalized).getSk()); Assertions.assertEquals(connector.getSessionToken(), S3ThriftAdapter.getS3TStorageParam(normalized).getToken()); Assertions.assertEquals(connector.getSecretKey(), CloudObjectStoreAdapter.getObjStoreInfoPB(normalized).getSk()); Assertions.assertEquals(789, S3ThriftAdapter.getS3TStorageParam(normalized).getConnTimeoutMs()); + Assertions.assertFalse(normalized.containsKey("gs.access_key")); Assertions.assertFalse(normalized.containsKey("gs.secret_key")); Assertions.assertFalse(normalized.containsKey("gs.session_token")); Assertions.assertFalse(normalized.containsKey("gs.connection.timeout")); - // Within the same ALTER input, gs.* also takes precedence over s3.*. + // Within the same ALTER input, gs.* still takes precedence over s3.*. resource.modifyProperties(ImmutableMap.of("gs.secret_key", "updated-gs-secret", S3ResourceCompat.SECRET_KEY, "ignored-s3-secret")); Assertions.assertEquals("updated-gs-secret", resource.getProperty(S3ResourceCompat.SECRET_KEY)); From dec2d8dd4696a42a81ade754e50bc0d8d536d28a Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Wed, 7 Oct 2026 20:45:26 +0800 Subject: [PATCH 04/47] [fix](cloud) Preserve vault credentials when clearing an unused IAM role ### What problem does this PR solve? Related PR: #68540 Problem Summary: Clearing s3.role_arn previously erased static keys and native GCP credentials before persisting the vault. Only replace other credentials when selecting a nonempty AWS role or an explicit AWS credential provider. Cover encrypted static credentials, native GCP credentials, and repeated clears at the RPC/persistence boundary. Also use TableNameInfo in ExportCommandTest and restore the role repository helper still used by the legacy AWS suite. ### Release note Clearing an unused AWS role preserves the vault's active static or GCP credentials. ### Check List (For Author) - Test: clang-format 16 check, build hygiene, Java Checkstyle (JDK 20), git diff --check, and Groovy helper smoke checks passed. Added cloud RPC/persistence regression coverage. FE UT and cloud build could not start because the configured JDK 17 lacks lib/modules; cloud third-party build dependencies and a live Doris/AWS environment are also unavailable. Full unit/integration tests were not run. - Behavior changed: Yes, clearing an unused role preserves existing credentials. - Does this need documentation: No --- .../meta-service/meta_service_resource.cpp | 12 ++++-- cloud/test/meta_service_test.cpp | 40 +++++++++++++++++++ .../plans/commands/ExportCommandTest.java | 6 +-- .../doris/regression/suite/Syncer.groovy | 23 +++++++++++ 4 files changed, 74 insertions(+), 7 deletions(-) diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index fbdaae334fd185..5b9ca4c04136a5 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -1186,10 +1186,14 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt } if (obj_info.has_role_arn() || obj_info.has_cred_provider_type()) { - new_vault.mutable_obj_info()->clear_ak(); - new_vault.mutable_obj_info()->clear_sk(); - new_vault.mutable_obj_info()->clear_encryption_info(); - new_vault.mutable_obj_info()->clear_credential(); + // Only selecting AWS authentication replaces other credentials. Clearing + // an unused role must preserve static keys and native GCP credentials. + if (!obj_info.role_arn().empty() || obj_info.has_cred_provider_type()) { + new_vault.mutable_obj_info()->clear_ak(); + new_vault.mutable_obj_info()->clear_sk(); + new_vault.mutable_obj_info()->clear_encryption_info(); + new_vault.mutable_obj_info()->clear_credential(); + } if (obj_info.has_role_arn()) { if (obj_info.role_arn().empty()) { diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index d5ef2b246d16ba..0cf8f1908912b9 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -780,6 +780,46 @@ TEST(MetaServiceTest, AlterS3StorageVaultTest) { ASSERT_TRUE(stored_vault.ParseFromString(val)); }; + // Clearing an unused AWS role must preserve the active credentials, + // including encryption metadata, through the RPC and persistence boundary. + for (const auto* vault_id : {"2", "3"}) { + StorageVaultPB stored; + get_test_vault(vault_id, stored); + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(stored.name()); + auto* update = req.mutable_vault()->mutable_obj_info(); + auto alter_vault = [&]() { + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, + nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + }; + if (stored.obj_info().has_ak()) { + update->set_ak("ak"); + update->set_sk("sk"); + alter_vault(); + get_test_vault(vault_id, stored); + ASSERT_TRUE(stored.obj_info().has_encryption_info()); + } else { + ASSERT_TRUE(stored.obj_info().credential().has_gcp_credential()); + } + stored.mutable_obj_info()->clear_mtime(); + const auto original = stored.obj_info().SerializeAsString(); + update->Clear(); + update->set_role_arn(""); + // Retrying the same field clear must also preserve the credentials. + for (int attempt = 0; attempt < 2; ++attempt) { + alter_vault(); + get_test_vault(vault_id, stored); + stored.mutable_obj_info()->clear_mtime(); + ASSERT_EQ(stored.obj_info().SerializeAsString(), original); + } + } + // A rejected credential update must not commit a rename that was applied // earlier in the request. { diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java index aefbe2900c6fff..53d24e4c935afd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java @@ -19,7 +19,7 @@ import org.apache.doris.analysis.BrokerDesc; import org.apache.doris.analysis.StorageBackend.StorageType; -import org.apache.doris.analysis.TableName; +import org.apache.doris.catalog.info.TableNameInfo; import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; import org.apache.doris.filesystem.auth.GcpCredential; @@ -138,10 +138,10 @@ private ExportJob generateExportJob(String path, BrokerDesc broker) throws Excep path, ImmutableMap.of(), Optional.of(broker)); // Inspect the job before registration so the test never starts an export task. Method method = ExportCommand.class.getDeclaredMethod("generateExportJob", - ConnectContext.class, Map.class, TableName.class); + ConnectContext.class, Map.class, TableNameInfo.class); method.setAccessible(true); return (ExportJob) method.invoke(command, connectContext, ImmutableMap.of(), - new TableName("internal", "export_test", "T1")); + new TableNameInfo("internal", "export_test", "T1")); } @Test diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy index 69ab664ec2e6b0..3a0b74d0ad555c 100644 --- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy +++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Syncer.groovy @@ -1027,6 +1027,29 @@ class Syncer { """ } + void createS3RepositoryWithRole(String name, boolean readOnly = false) { + String roleArn = suite.context.config.awsRoleArn + String externalId = suite.context.config.awsExternalId + String endpoint = suite.context.config.awsEndpoint + String region = suite.context.config.awsRegion + String bucket = suite.context.config.awsBucket + String prefix = suite.context.config.awsPrefix + + suite.try_sql "DROP REPOSITORY `${name}`" + suite.sql """ + CREATE ${readOnly ? "READ ONLY" : ""} REPOSITORY `${name}` + WITH S3 + ON LOCATION "s3://${bucket}/${prefix}/aws_iam_role_p0/${name}" + PROPERTIES + ( + "s3.endpoint" = "${endpoint}", + "s3.region" = "${region}", + "s3.role_arn" = "${roleArn}", + "s3.external_id" = "${externalId}" + ) + """ + } + void createObjectStorageIamRepository(String name, Map iamConfig, Map authCase, boolean readOnly = false) { suite.try_sql "DROP REPOSITORY `${name}`" suite.sql """ From 238355a0f0755c48222ceef6900cde0695bd49d6 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Wed, 7 Oct 2026 22:19:42 +0800 Subject: [PATCH 05/47] [fix](cloud) Clear native credentials when replacing instance access keys ### What problem does this PR solve? Related PR: #68540 Problem Summary: Legacy instance key updates retained native GCP credentials, so clients kept using OAuth after a successful AK/SK update. Clear the native envelope in both instance update helpers, including both user-id branches, and allow unchanged keys to repair an already persisted mixed auth state. Reject empty replacement keys before removing native authentication. Add RPC/persistence coverage for native-to-key transitions, mixed-state repair, and rejection without mutation. Guard optional AWS external-ID assertions so an omitted external ID does not fail the resource ALTER regression suite. ### Release note Replacing native GCP authentication with instance access keys now makes the new keys effective. AWS IAM resource regression tests accept an omitted external ID. ### Check List (For Author) - Test: Added regression coverage; compilation and test execution skipped at user request. Applied repository clang-format 16; git diff --check passed. - Behavior changed: Yes, instance key updates remove native authentication. - Does this need documentation: No --- .../meta-service/meta_service_resource.cpp | 28 ++++- cloud/test/meta_service_test.cpp | 116 +++++++++++++++++- .../test_alter_resource_with_role.groovy | 12 +- 3 files changed, 149 insertions(+), 7 deletions(-) diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index 5b9ca4c04136a5..889c63c35f5153 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -362,7 +362,14 @@ static int alter_instance_obj_store_info_by_id(InstanceInfoPB& instance, } if (role_arn.empty()) { - if (it.ak() == ak && it.sk() == sk) { + if (has_obj_credential(it) && (ak.empty() || sk.empty())) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = "nonempty access key and secret key are required to replace native " + "credentials"; + return -1; + } + // Resubmitting keys must also repair a previously persisted mixed auth state. + if (it.ak() == ak && it.sk() == sk && !has_obj_credential(it)) { code = MetaServiceCode::OK; msg = "ak/sk not changed"; return 1; @@ -370,6 +377,7 @@ static int alter_instance_obj_store_info_by_id(InstanceInfoPB& instance, it.clear_role_arn(); it.clear_external_id(); it.clear_cred_provider_type(); + it.clear_credential(); it.set_ak(std::string(ak)); it.set_sk(std::string(sk)); @@ -496,7 +504,13 @@ static int update_instance_ak_sk(InstanceInfoPB& instance, const UpdateAkSkReque proto_to_json(*request); return -1; } - if (it.ak() == ak && it.sk() == sk) { + if (has_obj_credential(it) && (ak.empty() || sk.empty())) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = "nonempty access key and secret key are required to replace native " + "credentials"; + return -1; + } + if (it.ak() == ak && it.sk() == sk && !has_obj_credential(it)) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "ak sk eq original, please check it"; return -1; @@ -505,6 +519,7 @@ static int update_instance_ak_sk(InstanceInfoPB& instance, const UpdateAkSkReque it.set_user_id(user_id); it.set_ak(ak); it.set_sk(sk); + it.clear_credential(); it.mutable_encryption_info()->CopyFrom(encryption_info); update_record << "update obj_info's ak sk without user_id, instance_id: " << instance.instance_id() << " obj_info_id: " << it.id() @@ -515,7 +530,13 @@ static int update_instance_ak_sk(InstanceInfoPB& instance, const UpdateAkSkReque } if (it.user_id() == user_id) { has_found_alter_obj_info = true; - if (it.ak() == ak && it.sk() == sk) { + if (has_obj_credential(it) && (ak.empty() || sk.empty())) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = "nonempty access key and secret key are required to replace native " + "credentials"; + return -1; + } + if (it.ak() == ak && it.sk() == sk && !has_obj_credential(it)) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "ak sk eq original, please check it"; return -1; @@ -523,6 +544,7 @@ static int update_instance_ak_sk(InstanceInfoPB& instance, const UpdateAkSkReque it.set_mtime(time); it.set_ak(ak); it.set_sk(sk); + it.clear_credential(); it.mutable_encryption_info()->CopyFrom(encryption_info); update_record << "update obj_info's ak sk, instance_id: " << instance.instance_id() << " obj_info_id: " << it.id() << " user_id: " << user_id diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index 0cf8f1908912b9..7dc428a5095f34 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -11468,7 +11468,8 @@ TEST(MetaServiceTest, UpdateAkSkTest) { std::string cipher_sk = "JUkuTDctR+ckJtnPkLScWaQZRcOtWBhsLLpnCRxQLxr734qB8cs6gNLH6grE1FxO"; std::string plain_sk = "Hx60p12123af234541nsVsffdfsdfghsdfhsdf34t"; - auto update = [&](bool with_user_id, bool with_wrong_user_id) { + auto update = [&](bool with_user_id, bool with_wrong_user_id, bool with_gcp_credential = false, + bool with_existing_keys = false) { std::unique_ptr txn; ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); std::string key; @@ -11482,6 +11483,20 @@ TEST(MetaServiceTest, UpdateAkSkTest) { } obj_info.set_ak("ak"); obj_info.set_sk("sk"); + if (with_gcp_credential) { + obj_info.set_provider(ObjectStoreInfoPB::GCP); + auto* credential = obj_info.mutable_credential()->mutable_gcp_credential(); + credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + credential->set_impersonation_service_account("test@project.iam.gserviceaccount.com"); + obj_info.clear_ak(); + obj_info.clear_sk(); + if (with_existing_keys) { + // Reproduce the mixed state persisted before this fix. Resubmitting + // the same keys must still remove the native credential. + obj_info.set_ak("new_ak"); + obj_info.set_sk(cipher_sk); + } + } InstanceInfoPB instance; instance.add_obj_info()->CopyFrom(obj_info); val = instance.SerializeAsString(); @@ -11500,12 +11515,34 @@ TEST(MetaServiceTest, UpdateAkSkTest) { ram_user.set_sk(plain_sk); req.add_internal_bucket_user()->CopyFrom(ram_user); + if (with_gcp_credential && !with_wrong_user_id) { + // Empty replacement keys must not erase the active native credential. + for (int empty_keys = 1; empty_keys <= 3; ++empty_keys) { + UpdateAkSkRequest invalid_req = req; + auto* keys = invalid_req.mutable_internal_bucket_user(0); + if (empty_keys & 1) keys->set_ak(""); + if (empty_keys & 2) keys->set_sk(""); + brpc::Controller invalid_cntl; + UpdateAkSkResponse invalid_res; + meta_service->update_ak_sk( + reinterpret_cast<::google::protobuf::RpcController*>(&invalid_cntl), + &invalid_req, &invalid_res, nullptr); + ASSERT_EQ(invalid_res.status().code(), MetaServiceCode::INVALID_ARGUMENT); + InstanceInfoPB unchanged; + get_test_instance(unchanged); + ASSERT_EQ(unchanged.obj_info(0).SerializeAsString(), obj_info.SerializeAsString()); + } + } + brpc::Controller cntl; UpdateAkSkResponse res; meta_service->update_ak_sk(reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); if (with_wrong_user_id) { ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT); + InstanceInfoPB unchanged; + get_test_instance(unchanged); + ASSERT_EQ(unchanged.obj_info(0).SerializeAsString(), obj_info.SerializeAsString()); } else { ASSERT_EQ(res.status().code(), MetaServiceCode::OK); InstanceInfoPB update_instance; @@ -11513,12 +11550,25 @@ TEST(MetaServiceTest, UpdateAkSkTest) { ASSERT_EQ(update_instance.obj_info(0).user_id(), "111"); ASSERT_EQ(update_instance.obj_info(0).ak(), "new_ak"); ASSERT_EQ(update_instance.obj_info(0).sk(), cipher_sk); + ASSERT_FALSE(update_instance.obj_info(0).has_credential()); + ASSERT_TRUE(update_instance.obj_info(0).has_encryption_info()); + if (with_gcp_credential) { + ASSERT_EQ(update_instance.obj_info(0).provider(), ObjectStoreInfoPB::GCP); + } } }; update(false, false); update(true, false); update(true, true); + update(false, false, true); + update(true, false, true); + update(true, true, true); + update(false, false, true, true); + update(true, false, true, true); + + sp->disable_processing(); + sp->clear_all_call_backs(); } TEST(MetaServiceTest, AlterIamTest) { @@ -12716,6 +12766,70 @@ TEST(MetaServiceTest, AlterObjInfoTest) { ASSERT_FALSE(instance.obj_info(0).has_external_id()); } + // Both legacy ALTER operations must replace native GCP authentication, + // including a mixed state left by an earlier key update using the same keys. + for (auto op : {AlterObjStoreInfoRequest::ALTER_OBJ_INFO, + AlterObjStoreInfoRequest::LEGACY_UPDATE_AK_SK}) { + for (bool with_existing_keys : {false, true}) { + InstanceInfoPB native_instance; + auto* obj = native_instance.add_obj_info(); + obj->set_id("1"); + obj->set_provider(ObjectStoreInfoPB::GCP); + auto* credential = obj->mutable_credential()->mutable_gcp_credential(); + credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + credential->set_impersonation_service_account("test@project.iam.gserviceaccount.com"); + if (with_existing_keys) { + obj->set_ak("new_ak"); + obj->set_sk(cipher_sk); + } + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + txn->put(key, native_instance.SerializeAsString()); + ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK); + + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(op); + req.mutable_obj()->set_id("1"); + req.mutable_obj()->set_ak("new_ak"); + req.mutable_obj()->set_sk(plain_sk); + // A role-only clear and either empty key must leave the stored + // native credential intact, including when legacy keys also exist. + for (int empty_keys = 0; empty_keys <= 3; ++empty_keys) { + AlterObjStoreInfoRequest invalid_req = req; + if (empty_keys == 0) { + invalid_req.mutable_obj()->clear_ak(); + invalid_req.mutable_obj()->clear_sk(); + invalid_req.mutable_obj()->set_role_arn(""); + } else { + if (empty_keys & 1) invalid_req.mutable_obj()->set_ak(""); + if (empty_keys & 2) invalid_req.mutable_obj()->set_sk(""); + } + brpc::Controller invalid_cntl; + AlterObjStoreInfoResponse invalid_res; + meta_service->alter_obj_store_info( + reinterpret_cast<::google::protobuf::RpcController*>(&invalid_cntl), + &invalid_req, &invalid_res, nullptr); + ASSERT_EQ(invalid_res.status().code(), MetaServiceCode::INVALID_ARGUMENT); + InstanceInfoPB unchanged; + get_test_instance(unchanged); + ASSERT_EQ(unchanged.obj_info(0).SerializeAsString(), obj->SerializeAsString()); + } + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_obj_store_info( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, + nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + InstanceInfoPB stored; + get_test_instance(stored); + ASSERT_EQ(stored.obj_info(0).provider(), ObjectStoreInfoPB::GCP); + ASSERT_EQ(stored.obj_info(0).ak(), "new_ak"); + ASSERT_EQ(stored.obj_info(0).sk(), cipher_sk); + ASSERT_TRUE(stored.obj_info(0).has_encryption_info()); + ASSERT_FALSE(stored.obj_info(0).has_credential()); + } + } + SyncPoint::get_instance()->disable_processing(); SyncPoint::get_instance()->clear_all_call_backs(); } diff --git a/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy b/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy index 28793ec085e9c9..cdf3fb5cf578eb 100644 --- a/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy +++ b/regression-test/suites/object_storage_iam_p0/test_alter_resource_with_role.groovy @@ -96,7 +96,9 @@ suite("test_alter_resource_with_role") { def result = sql """ SHOW RESOURCES WHERE NAME = "${resourceName}"; """ log.info("result:${result}") assertTrue(!result.toString().contains(roleArn)) - assertTrue(!result.toString().contains(externalId)); + if (!externalId.isEmpty()) { + assertTrue(!result.toString().contains(externalId)) + } assertTrue(result.toString().contains("error_ak")); // data_sizes is one arrayList, t is tablet @@ -158,7 +160,9 @@ suite("test_alter_resource_with_role") { result = sql """ SHOW RESOURCES WHERE NAME = "${resourceName}"; """ log.info("result:${result}") assertTrue(result.toString().contains(roleArn)) - assertTrue(result.toString().contains(externalId)); + if (!externalId.isEmpty()) { + assertTrue(result.toString().contains(externalId)) + } assertTrue(!result.toString().contains("error_ak")); // report_tablet_interval_seconds = 60 sleep wait for report policy @@ -200,7 +204,9 @@ suite("test_alter_resource_with_role") { result = sql """ SHOW RESOURCES WHERE NAME = "${resourceName}"; """ log.info("result:${result}") assertTrue(!result.toString().contains(roleArn)) - assertTrue(!result.toString().contains(externalId)); + if (!externalId.isEmpty()) { + assertTrue(!result.toString().contains(externalId)) + } assertTrue(result.toString().contains(awsAccessKey)); // report_tablet_interval_seconds = 60 sleep wait for report policy From 334466438a94778559b035451ab4716841e3bfef Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 10:53:20 +0800 Subject: [PATCH 06/47] [fix](storage) Restrict GCP OAuth endpoints and repair storage updates ### What problem does this PR solve? Related PR: #68540 Problem Summary: Native GCP clients attached process bearer tokens to arbitrary storage endpoints. Require Google Cloud Storage HTTPS endpoints on port 443 in FE properties and request interception, and in the shared BE/Cloud client before fetching tokens. Keep TLS verification enabled, disable native-client redirects, and reject AWS access-point ARNs in GCS bucket fields. HMAC and anonymous endpoint handling retain their existing behavior. Exclude the Doris cluster Auth-Token header from IAM signBlob and metadata HTTP requests. Admit validated native credentials for legacy ADD_OBJ_INFO requests, preserve role-only ADD_S3_VAULT requests with an omitted provider type, and pass the locked CA-path snapshot to both GCP consumers. ### Release note Native GCP OAuth requires a Google Cloud Storage HTTPS endpoint on port 443; untrusted custom endpoints and plaintext endpoints are rejected. Legacy object storage ADD accepts native GCP credentials, and role-only vault ADD continues to default to INSTANCE_PROFILE. ### Check List (For Author) - Test: Added FE, BE, and Cloud unit coverage. Java Checkstyle, clang-format 16, build hygiene, and git diff --check passed. An isolated check compiled the actual C++ endpoint validator and passed 14 endpoint cases. Full unit tests and BE/Cloud builds were not run: standard FE/BE test entry points stopped because JDK 17 is unavailable, and this checkout has no thirdparty bundle. - Behavior changed: Yes, endpoint restrictions and corrected credential admission. - Does this need documentation: Yes, native OAuth endpoint restrictions are described in the release note above. --- be/src/io/fs/gcs_signed_url_provider.cpp | 5 +- be/src/service/http/http_client.cpp | 7 +- be/src/service/http/http_client.h | 5 +- be/src/util/s3_util.cpp | 4 +- be/test/io/s3_client_factory_test.cpp | 49 +++++++++- be/test/service/http/http_client_test.cpp | 22 +++++ .../meta-service/meta_service_resource.cpp | 26 ++++-- cloud/test/meta_service_test.cpp | 89 +++++++++++++++++++ common/cpp/obj-client/auth/gcp/gcp_auth.cpp | 9 ++ common/cpp/obj-client/auth/gcp/gcp_auth.h | 1 + .../cpp/obj-client/auth/gcp/gcp_s3_client.cpp | 63 ++++++++++--- .../cpp/obj-client/auth/gcp/gcp_s3_client.h | 5 +- .../apache/doris/filesystem/gcs/GcpAuth.java | 30 ++++++- .../gcs/GcsFileSystemProperties.java | 3 + .../doris/filesystem/gcs/GcpAuthTest.java | 16 ++++ .../gcs/GcsFileSystemPropertiesTest.java | 18 +++- 16 files changed, 315 insertions(+), 37 deletions(-) diff --git a/be/src/io/fs/gcs_signed_url_provider.cpp b/be/src/io/fs/gcs_signed_url_provider.cpp index d69e0fa46e5bf4..22eddad6dcd6d9 100644 --- a/be/src/io/fs/gcs_signed_url_provider.cpp +++ b/be/src/io/fs/gcs_signed_url_provider.cpp @@ -101,7 +101,7 @@ Status call_iam_sign_blob(std::string_view access_token, std::string_view servic HttpClient client; // Keep the response body for non-2xx replies so IAM permission and // service-account errors are actionable to operators. - RETURN_IF_ERROR(client.init(endpoint, false)); + RETURN_IF_ERROR(client.init(endpoint, false, HttpClient::AuthTokenMode::NONE)); client.set_authorization("Bearer " + std::string(access_token)); client.set_content_type("application/json"); client.set_timeout_ms(request_timeout_ms > 0 ? request_timeout_ms : 10000); @@ -133,7 +133,8 @@ Status fetch_metadata_service_account_email(int64_t request_timeout_ms, std::str : DEFAULT_METADATA_HOST; HttpClient client; RETURN_IF_ERROR(client.init( - fmt::format("http://{}{}", metadata_host, METADATA_SERVICE_ACCOUNT_EMAIL_PATH))); + fmt::format("http://{}{}", metadata_host, METADATA_SERVICE_ACCOUNT_EMAIL_PATH), true, + HttpClient::AuthTokenMode::NONE)); client.set_header("Metadata-Flavor", "Google"); client.set_timeout_ms(request_timeout_ms > 0 ? std::min(request_timeout_ms, METADATA_REQUEST_TIMEOUT_MS) diff --git a/be/src/service/http/http_client.cpp b/be/src/service/http/http_client.cpp index 7a82549f78ea52..59c5e95211ff6b 100644 --- a/be/src/service/http/http_client.cpp +++ b/be/src/service/http/http_client.cpp @@ -281,7 +281,8 @@ HttpClient::~HttpClient() { } } -Status HttpClient::init(const std::string& url, bool set_fail_on_error) { +Status HttpClient::init(const std::string& url, bool set_fail_on_error, + AuthTokenMode auth_token_mode) { if (_curl == nullptr) { _curl = curl_easy_init(); if (_curl == nullptr) { @@ -358,9 +359,11 @@ Status HttpClient::init(const std::string& url, bool set_fail_on_error) { return Status::InternalError("fail to set CURLOPT_URL"); } + if (auth_token_mode == AuthTokenMode::CLUSTER) { #ifndef BE_TEST - set_auth_token(ExecEnv::GetInstance()->cluster_info()->curr_auth_token); + set_auth_token(ExecEnv::GetInstance()->cluster_info()->curr_auth_token); #endif + } return Status::OK(); } diff --git a/be/src/service/http/http_client.h b/be/src/service/http/http_client.h index cdeaf9a4d342f4..b79a9a6c14ce90 100644 --- a/be/src/service/http/http_client.h +++ b/be/src/service/http/http_client.h @@ -35,6 +35,8 @@ namespace doris { // Helper class to access HTTP resource class HttpClient { public: + enum class AuthTokenMode { CLUSTER, NONE }; + HttpClient(); ~HttpClient(); @@ -47,7 +49,8 @@ class HttpClient { // this function must call before other function, // you can call this multiple times to reuse this object - Status init(const std::string& url, bool set_fail_on_error = true); + Status init(const std::string& url, bool set_fail_on_error = true, + AuthTokenMode auth_token_mode = AuthTokenMode::CLUSTER); void set_method(HttpMethod method); diff --git a/be/src/util/s3_util.cpp b/be/src/util/s3_util.cpp index 29c7b15ff3111a..5904d399f645fe 100644 --- a/be/src/util/s3_util.cpp +++ b/be/src/util/s3_util.cpp @@ -442,7 +442,7 @@ Result> S3ClientFactory::_create_s3_client .payload_signing_policy = Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, .use_virtual_addressing = s3_conf.use_virtual_addressing, - .ca_cert_path = _ca_cert_file_path, + .ca_cert_path = ca_cert_file_path, }); std::shared_ptr gcs_signing_token_provider; @@ -453,7 +453,7 @@ Result> S3ClientFactory::_create_s3_client auto signing_caller_credential = *gcp_credential; signing_caller_credential.impersonation_service_account.clear(); gcs_signing_token_provider = std::make_shared( - signing_caller_credential, _ca_cert_file_path, GcpTokenScope::CloudPlatform); + signing_caller_credential, ca_cert_file_path, GcpTokenScope::CloudPlatform); } S3ObjStorageClient::SignedUrlGenerator signed_url_generator; diff --git a/be/test/io/s3_client_factory_test.cpp b/be/test/io/s3_client_factory_test.cpp index a1cfde54e77e8f..097efcb7665d30 100644 --- a/be/test/io/s3_client_factory_test.cpp +++ b/be/test/io/s3_client_factory_test.cpp @@ -172,17 +172,28 @@ S3ClientConf make_gcp_native_conf(std::string impersonation_service_account = {} class FixedTokenGcpS3Client final : public GcpS3Client { public: - FixedTokenGcpS3Client() - : GcpS3Client(GcpCredentialConfig {}, "", Aws::Client::ClientConfiguration {}, + explicit FixedTokenGcpS3Client(std::string endpoint = "https://storage.googleapis.com") + : GcpS3Client(GcpCredentialConfig {}, "", make_config(endpoint), Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true) {} using GcpS3Client::BuildHttpRequest; using GcpS3Client::authorize_request; std::optional token = "test-gcp-token"; + mutable int token_requests = 0; protected: - std::optional fetch_token() const override { return token; } + std::optional fetch_token() const override { + ++token_requests; + return token; + } + +private: + static Aws::Client::ClientConfiguration make_config(const std::string& endpoint) { + Aws::Client::ClientConfiguration config; + config.endpointOverride = endpoint; + return config; + } }; class SyncPointProcessingGuard { @@ -407,7 +418,7 @@ TEST_F(S3ClientFactoryTest, GcpBearerHeaderSurvivesAnonymousAwsSigning) { Aws::Http::HttpMethod::HTTP_GET, Aws::Utils::Stream::DefaultResponseStreamFactoryMethod); - ASSERT_FALSE(client.authorize_request(request).has_value()); + ASSERT_FALSE(client.authorize_request(request, request.GetBucket()).has_value()); client.BuildHttpRequest(request, http_request); EXPECT_EQ(http_request->GetHeaderValue("Authorization"), "Bearer test-gcp-token"); @@ -420,6 +431,36 @@ TEST_F(S3ClientFactoryTest, GcpBearerHeaderSurvivesAnonymousAwsSigning) { EXPECT_EQ(http_request->GetHeaderValue("Authorization"), "Bearer test-gcp-token"); } +TEST_F(S3ClientFactoryTest, GcpRejectsUntrustedEndpointsBeforeFetchingToken) { + for (const auto* endpoint : {"https://attacker.example", "http://storage.googleapis.com", + "https://storage.googleapis.com.attacker.example", + "https://storage.googleapis.com@attacker.example", + "https://storage.googleapis.com:8443", ""}) { + FixedTokenGcpS3Client client(endpoint); + Aws::S3::Model::GetObjectRequest request; + request.SetBucket("bucket"); + request.SetKey("key"); + auto outcome = client.GetObject(request); + ASSERT_FALSE(outcome.IsSuccess()) << endpoint; + EXPECT_EQ(outcome.GetError().GetExceptionName(), "GcpEndpointError"); + EXPECT_EQ(client.token_requests, 0); + } + FixedTokenGcpS3Client client; + Aws::S3::Model::GetObjectRequest request; + request.SetBucket("arn:aws:s3:us-east-1:123456789012:accesspoint/example"); + EXPECT_FALSE(client.GetObject(request).IsSuccess()); + EXPECT_EQ(client.token_requests, 0); +} + +TEST_F(S3ClientFactoryTest, GcpAcceptsGoogleStorageEndpoints) { + for (const auto* endpoint : + {"https://storage.googleapis.com", "https://bucket.storage.googleapis.com:443/", + "https://us-central1-storage.googleapis.com", + "https://storage.us-central1.rep.googleapis.com"}) { + EXPECT_TRUE(is_valid_gcp_storage_endpoint(endpoint)) << endpoint; + } +} + TEST_F(S3ClientFactoryTest, GcpTokenFailureRejectsEveryObjectStorageOperation) { FixedTokenGcpS3Client client; for (const auto& token : {std::optional {}, std::optional {""}}) { diff --git a/be/test/service/http/http_client_test.cpp b/be/test/service/http/http_client_test.cpp index 06d23a64185ddd..36f576b1d0f86e 100644 --- a/be/test/service/http/http_client_test.cpp +++ b/be/test/service/http/http_client_test.cpp @@ -97,6 +97,14 @@ class HttpClientTestSimplePostHandler : public HttpHandler { } }; +class ExternalAuthHandler : public HttpHandler { +public: + void handle(HttpRequest* req) override { + HttpChannel::send_reply(req, req->header(HttpHeaders::AUTH_TOKEN) + "|" + + req->header(HttpHeaders::AUTHORIZATION)); + } +}; + class HttpNotFoundHandler : public HttpHandler { public: void handle(HttpRequest* req) override { @@ -135,6 +143,7 @@ constexpr std::string_view TMP_DIR = "./http_test_tmp"; static HttpClientTestSimpleGetHandler s_simple_get_handler; static HttpClientTestSimplePostHandler s_simple_post_handler; static HttpNotFoundHandler s_not_found_handler; +static ExternalAuthHandler s_external_auth_handler; static HttpDownloadFileHandler s_download_file_handler; static HttpBatchDownloadFileHandler s_batch_download_file_handler; @@ -149,6 +158,7 @@ class HttpClientTest : public testing::Test { s_server->register_handler(HEAD, "/simple_get", &s_simple_get_handler); s_server->register_handler(POST, "/simple_post", &s_simple_post_handler); s_server->register_handler(GET, "/not_found", &s_not_found_handler); + s_server->register_handler(GET, "/external_auth", &s_external_auth_handler); s_server->register_handler(HEAD, "/download_file", &s_download_file_handler); s_server->register_handler(HEAD, "/api/_tablet/_batch_download", &s_batch_download_file_handler); @@ -190,6 +200,18 @@ TEST_F(HttpClientTest, get_normal) { EXPECT_EQ(5, len); } +TEST_F(HttpClientTest, external_request_omits_cluster_token) { + HttpClient client; + ASSERT_TRUE(client.init(hostname + "/external_auth").ok()); + client.set_auth_token("previous-cluster-token"); + ASSERT_TRUE( + client.init(hostname + "/external_auth", false, HttpClient::AuthTokenMode::NONE).ok()); + client.set_authorization("Bearer google-token"); + std::string response; + ASSERT_TRUE(client.execute(&response).ok()); + EXPECT_EQ(response, "|Bearer google-token"); +} + TEST_F(HttpClientTest, download) { HttpClient client; auto st = client.init(hostname + "/simple_get"); diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index 889c63c35f5153..a78dd505ecb23b 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -1341,9 +1341,20 @@ static int extract_object_storage_info(const AlterObjStoreInfoRequest* request, auto& [ak, sk, bucket, prefix, endpoint, external_endpoint, region, use_path_style, role_arn, external_id] = obj_desc; - bool use_credential_provider_for_add_vault = - request->op() == AlterObjStoreInfoRequest::ADD_S3_VAULT && use_credential_provider(obj); - if (!obj.has_role_arn() && !use_credential_provider_for_add_vault) { + bool native_obj_add = + request->op() == AlterObjStoreInfoRequest::ADD_OBJ_INFO && has_obj_credential(obj); + if (native_obj_add) { + if (auto error = validate_obj_credential(obj); error.has_value()) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = *error; + return -1; + } + } + bool use_credential_provider_for_add = + (request->op() == AlterObjStoreInfoRequest::ADD_S3_VAULT && + use_credential_provider(obj)) || + native_obj_add; + if (!obj.has_role_arn() && !use_credential_provider_for_add) { if (!obj.has_ak() || !obj.has_sk()) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "s3 obj info err " + proto_to_json(*request); @@ -1582,8 +1593,9 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr } if (use_credential_provider(obj)) { - bool valid_aws = obj.has_cred_provider_type() && !has_obj_credential(obj) && - obj.has_provider() && obj.provider() == ObjectStoreInfoPB::S3; + bool valid_aws = (obj.has_cred_provider_type() || has_non_empty_role_arn(obj)) && + !has_obj_credential(obj) && obj.has_provider() && + obj.provider() == ObjectStoreInfoPB::S3; bool valid_credential = has_obj_credential(obj) && !validate_obj_credential(obj).has_value(); if (!valid_aws && !valid_credential) { @@ -1989,8 +2001,8 @@ void MetaServiceImpl::alter_obj_store_info(google::protobuf::RpcController* cont return; } // ATTN: prefix may be empty - if (((ak.empty() || sk.empty()) && role_arn.empty()) || bucket.empty() || - endpoint.empty() || region.empty() || prefix.empty()) { + if (((ak.empty() || sk.empty()) && role_arn.empty() && !has_obj_credential(obj)) || + bucket.empty() || endpoint.empty() || region.empty() || prefix.empty()) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "s3 conf info err, please check it"; return; diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index 244064e273328c..d28ca5d62310ea 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -5029,6 +5029,95 @@ TEST(MetaServiceTest, CalcSyncVersionsTest) { } } +TEST(MetaServiceTest, NativeGcpLegacyAddReadback) { + auto meta_service = get_meta_service(); + brpc::Controller cntl; + std::string key; + instance_key({"test_instance"}, &key); + InstanceInfoPB instance; + instance.set_instance_id("test_instance"); + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + txn->put(key, instance.SerializeAsString()); + ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK); + + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ADD_OBJ_INFO); + auto* obj = req.mutable_obj(); + obj->set_provider(ObjectStoreInfoPB::GCP); + obj->set_bucket("native-bucket"); + obj->set_prefix("instance-prefix"); + obj->set_endpoint("storage.googleapis.com"); + obj->set_region("us-east1"); + auto* credential = obj->mutable_credential()->mutable_gcp_credential(); + credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); + credential->set_impersonation_service_account("target@test.iam.gserviceaccount.com"); + + // Reject mixed authentication before persisting anything. + obj->set_ak("unexpected-key"); + AlterObjStoreInfoResponse rejected; + meta_service->alter_obj_store_info(&cntl, &req, &rejected, nullptr); + ASSERT_EQ(rejected.status().code(), MetaServiceCode::INVALID_ARGUMENT); + obj->clear_ak(); + AlterObjStoreInfoResponse res; + meta_service->alter_obj_store_info(&cntl, &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + txn.reset(); + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(key, &val), TxnErrorCode::TXN_OK); + ASSERT_TRUE(instance.ParseFromString(val)); + ASSERT_EQ(instance.obj_info_size(), 1); + ASSERT_EQ(instance.obj_info(0).credential().SerializeAsString(), + obj->credential().SerializeAsString()); + ASSERT_FALSE(instance.obj_info(0).has_ak()); + ASSERT_FALSE(instance.obj_info(0).has_encryption_info()); +} + +TEST(MetaServiceTest, AddRoleOnlyVaultDefaultsToInstanceProfile) { + auto meta_service = get_meta_service(); + brpc::Controller cntl; + std::string key; + instance_key({"test_instance"}, &key); + InstanceInfoPB instance; + instance.set_instance_id("test_instance"); + instance.set_enable_storage_vault(true); + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + txn->put(key, instance.SerializeAsString()); + ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK); + + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ADD_S3_VAULT); + req.mutable_vault()->set_name("role_only_vault"); + auto* obj = req.mutable_obj(); + obj->set_provider(ObjectStoreInfoPB::S3); + obj->set_bucket("bucket"); + obj->set_endpoint("s3.us-east-1.amazonaws.com"); + obj->set_region("us-east-1"); + obj->set_role_arn("arn:aws:iam::123456789012:role/test"); + ASSERT_FALSE(obj->has_cred_provider_type()); + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault(&cntl, &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + txn.reset(); + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(key, &val), TxnErrorCode::TXN_OK); + ASSERT_TRUE(instance.ParseFromString(val)); + ASSERT_EQ(instance.resource_ids_size(), 1); + ASSERT_EQ(txn->get(storage_vault_key({"test_instance", instance.resource_ids(0)}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB vault; + ASSERT_TRUE(vault.ParseFromString(val)); + EXPECT_EQ(vault.obj_info().role_arn(), obj->role_arn()); + EXPECT_EQ(vault.obj_info().cred_provider_type(), CredProviderTypePB::INSTANCE_PROFILE); +} + TEST(MetaServiceTest, StageTest) { auto meta_service = get_meta_service(); brpc::Controller cntl; diff --git a/common/cpp/obj-client/auth/gcp/gcp_auth.cpp b/common/cpp/obj-client/auth/gcp/gcp_auth.cpp index 801b7b5c7f6836..f0af88056071c5 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_auth.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_auth.cpp @@ -19,10 +19,19 @@ #include #include +#include #include namespace doris { +bool is_valid_gcp_storage_endpoint(std::string_view endpoint) { + // Match the FE GcpAuth policy: Google-owned XML API hosts, HTTPS and port 443 only. + static const std::regex pattern( + R"(https://([a-z0-9-]+\.)*(storage\.googleapis\.com|[a-z0-9-]+-storage\.googleapis\.com|storage\.[a-z0-9-]+\.rep\.googleapis\.com)(:443)?/?)", + std::regex::icase); + return std::regex_match(endpoint.begin(), endpoint.end(), pattern); +} + bool is_valid_gcp_service_account_email(std::string_view email) { constexpr std::string_view user_managed_suffix = ".iam.gserviceaccount.com"; constexpr std::string_view compute_engine_domain = "developer.gserviceaccount.com"; diff --git a/common/cpp/obj-client/auth/gcp/gcp_auth.h b/common/cpp/obj-client/auth/gcp/gcp_auth.h index 99b35f7be5049b..8f05d7a580a44d 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_auth.h +++ b/common/cpp/obj-client/auth/gcp/gcp_auth.h @@ -51,6 +51,7 @@ struct GcpCredentialParseResult { }; bool is_valid_gcp_service_account_email(std::string_view email); +bool is_valid_gcp_storage_endpoint(std::string_view endpoint); std::optional parse_gcp_credential_provider_type( std::string_view provider_type); std::optional validate_gcp_credential(const GcpCredentialConfig& credential); diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp index da2a0fd9b7e1d7..8b40e666453562 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp @@ -43,20 +43,55 @@ #include #include +#include #include namespace doris { +namespace { + +Aws::Client::ClientConfiguration gcp_http_config(Aws::Client::ClientConfiguration config) { + // Do not forward bearer headers on a redirect or allow TLS verification to be disabled. + config.followRedirects = Aws::Client::FollowRedirectsPolicy::NEVER; + config.verifySSL = true; + config.scheme = Aws::Http::Scheme::HTTPS; + return config; +} + +bool trusted_endpoint(const Aws::Client::ClientConfiguration& config) { + std::string endpoint(config.endpointOverride.c_str()); + if (endpoint.find("://") == std::string::npos) { + endpoint = "https://" + endpoint; + } + return is_valid_gcp_storage_endpoint(endpoint); +} + +} // namespace GcpS3Client::GcpS3Client(const GcpCredentialConfig& credential, std::string ca_cert_path, Aws::Client::ClientConfiguration aws_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy payload_signing_policy, bool use_virtual_addressing) : Aws::S3::S3Client(std::make_shared(), - std::move(aws_config), payload_signing_policy, use_virtual_addressing), + gcp_http_config(aws_config), payload_signing_policy, + use_virtual_addressing), + _trusted_endpoint(trusted_endpoint(aws_config)), _token_provider(credential, ca_cert_path) {} std::optional GcpS3Client::authorize_request( - Aws::AmazonWebServiceRequest& request) const { + Aws::AmazonWebServiceRequest& request, std::string_view bucket) const { + // GCS bucket names cannot contain ARN, URL or authority delimiters. In particular, + // do not let the AWS endpoint resolver route an access-point ARN to an AWS host. + bool valid_bucket = std::all_of(bucket.begin(), bucket.end(), [](unsigned char c) { + return (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.'; + }); + if (!_trusted_endpoint || !valid_bucket) { + Aws::S3::S3Error error(Aws::S3::S3Errors::ACCESS_DENIED, "GcpEndpointError", + "Native GCP OAuth requires a trusted Google Cloud Storage HTTPS " + "endpoint on port 443 and a GCS bucket name", + false); + error.SetResponseCode(Aws::Http::HttpResponseCode::FORBIDDEN); + return error; + } auto token = fetch_token(); if (!token.has_value() || token->empty()) { // Return through the SDK outcome path before any object-storage HTTP request. @@ -74,7 +109,7 @@ std::optional GcpS3Client::authorize_request( Aws::S3::Model::AbortMultipartUploadOutcome GcpS3Client::AbortMultipartUpload( const Aws::S3::Model::AbortMultipartUploadRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::AbortMultipartUpload(authorized_request); @@ -83,7 +118,7 @@ Aws::S3::Model::AbortMultipartUploadOutcome GcpS3Client::AbortMultipartUpload( Aws::S3::Model::CompleteMultipartUploadOutcome GcpS3Client::CompleteMultipartUpload( const Aws::S3::Model::CompleteMultipartUploadRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::CompleteMultipartUpload(authorized_request); @@ -92,7 +127,7 @@ Aws::S3::Model::CompleteMultipartUploadOutcome GcpS3Client::CompleteMultipartUpl Aws::S3::Model::CreateMultipartUploadOutcome GcpS3Client::CreateMultipartUpload( const Aws::S3::Model::CreateMultipartUploadRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::CreateMultipartUpload(authorized_request); @@ -101,7 +136,7 @@ Aws::S3::Model::CreateMultipartUploadOutcome GcpS3Client::CreateMultipartUpload( Aws::S3::Model::DeleteObjectOutcome GcpS3Client::DeleteObject( const Aws::S3::Model::DeleteObjectRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::DeleteObject(authorized_request); @@ -110,7 +145,7 @@ Aws::S3::Model::DeleteObjectOutcome GcpS3Client::DeleteObject( Aws::S3::Model::DeleteObjectsOutcome GcpS3Client::DeleteObjects( const Aws::S3::Model::DeleteObjectsRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::DeleteObjects(authorized_request); @@ -119,7 +154,7 @@ Aws::S3::Model::DeleteObjectsOutcome GcpS3Client::DeleteObjects( Aws::S3::Model::GetBucketLifecycleConfigurationOutcome GcpS3Client::GetBucketLifecycleConfiguration( const Aws::S3::Model::GetBucketLifecycleConfigurationRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::GetBucketLifecycleConfiguration(authorized_request); @@ -128,7 +163,7 @@ Aws::S3::Model::GetBucketLifecycleConfigurationOutcome GcpS3Client::GetBucketLif Aws::S3::Model::GetBucketVersioningOutcome GcpS3Client::GetBucketVersioning( const Aws::S3::Model::GetBucketVersioningRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::GetBucketVersioning(authorized_request); @@ -137,7 +172,7 @@ Aws::S3::Model::GetBucketVersioningOutcome GcpS3Client::GetBucketVersioning( Aws::S3::Model::GetObjectOutcome GcpS3Client::GetObject( const Aws::S3::Model::GetObjectRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::GetObject(authorized_request); @@ -146,7 +181,7 @@ Aws::S3::Model::GetObjectOutcome GcpS3Client::GetObject( Aws::S3::Model::HeadObjectOutcome GcpS3Client::HeadObject( const Aws::S3::Model::HeadObjectRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::HeadObject(authorized_request); @@ -155,7 +190,7 @@ Aws::S3::Model::HeadObjectOutcome GcpS3Client::HeadObject( Aws::S3::Model::ListObjectsV2Outcome GcpS3Client::ListObjectsV2( const Aws::S3::Model::ListObjectsV2Request& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::ListObjectsV2(authorized_request); @@ -164,7 +199,7 @@ Aws::S3::Model::ListObjectsV2Outcome GcpS3Client::ListObjectsV2( Aws::S3::Model::PutObjectOutcome GcpS3Client::PutObject( const Aws::S3::Model::PutObjectRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::PutObject(authorized_request); @@ -173,7 +208,7 @@ Aws::S3::Model::PutObjectOutcome GcpS3Client::PutObject( Aws::S3::Model::UploadPartOutcome GcpS3Client::UploadPart( const Aws::S3::Model::UploadPartRequest& request) const { auto authorized_request = request; - if (auto error = authorize_request(authorized_request)) { + if (auto error = authorize_request(authorized_request, request.GetBucket())) { return std::move(*error); } return Aws::S3::S3Client::UploadPart(authorized_request); diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.h b/common/cpp/obj-client/auth/gcp/gcp_s3_client.h index 8b301dfc483e63..8d880ebf60f97f 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_s3_client.h +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.h @@ -24,6 +24,7 @@ #include #include +#include #include "cpp/obj-client/auth/gcp/gcp_auth.h" #include "cpp/obj-client/auth/gcp/gcp_token_provider.h" @@ -78,12 +79,14 @@ class GcpS3Client : public Aws::S3::S3Client { const Aws::S3::Model::UploadPartRequest& request) const override; protected: - std::optional authorize_request(Aws::AmazonWebServiceRequest& request) const; + std::optional authorize_request(Aws::AmazonWebServiceRequest& request, + std::string_view bucket) const; // Test seam for asserting the outgoing header without relying on ADC or metadata. virtual std::optional fetch_token() const; private: + const bool _trusted_endpoint; GcpTokenProvider _token_provider; }; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java index 9c1a38f48d1907..75cb284d6b0da1 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java @@ -45,6 +45,7 @@ import java.util.Collections; import java.util.List; import java.util.Map; +import java.util.regex.Pattern; /** Creates GCP OAuth2 credentials and adapts them to the S3-compatible client. */ public final class GcpAuth { @@ -52,6 +53,11 @@ public final class GcpAuth { "https://www.googleapis.com/auth/devstorage.read_write"; private static final String CLOUD_PLATFORM_SCOPE = "https://www.googleapis.com/auth/cloud-platform"; private static final int IMPERSONATION_LIFETIME_SECONDS = 3600; + // Global, locational and regional XML API endpoints, including virtual-hosted buckets. + private static final Pattern STORAGE_HOST = Pattern.compile( + "(?:[a-z0-9-]+\\.)*(?:storage\\.googleapis\\.com|" + + "[a-z0-9-]+-storage\\.googleapis\\.com|storage\\.[a-z0-9-]+\\.rep\\.googleapis\\.com)", + Pattern.CASE_INSENSITIVE); private GcpAuth() { } @@ -79,8 +85,26 @@ public static S3AsyncClient createS3AsyncClient(GcsFileSystemProperties properti } private static URI endpoint(GcsFileSystemProperties properties) { - String endpoint = properties.getEndpoint(); - return URI.create(endpoint.contains("://") ? endpoint : "https://" + endpoint); + return validateEndpoint(properties.getEndpoint()); + } + + static URI validateEndpoint(String endpoint) { + URI uri = URI.create(endpoint.contains("://") ? endpoint : "https://" + endpoint); + validateRequestUri(uri); + if (uri.getRawQuery() != null || (!uri.getRawPath().isEmpty() && !"/".equals(uri.getRawPath()))) { + throw new IllegalArgumentException("Native GCP OAuth endpoint must not contain a path or query"); + } + return uri; + } + + private static void validateRequestUri(URI uri) { + if (!"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null + || !STORAGE_HOST.matcher(uri.getHost()).matches() + || (uri.getPort() != -1 && uri.getPort() != 443) + || uri.getRawUserInfo() != null || uri.getRawFragment() != null) { + throw new IllegalArgumentException( + "Native GCP OAuth requires a trusted Google Cloud Storage HTTPS endpoint on port 443"); + } } private static S3Configuration serviceConfiguration(GcsFileSystemProperties properties) { @@ -130,6 +154,8 @@ public static ExecutionInterceptor bearerTokenInterceptor(GoogleCredentials cred public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) { try { + // Validate the resolved request host before obtaining or attaching process credentials. + validateRequestUri(context.httpRequest().getUri()); Map> metadata = credentials.getRequestMetadata(); List authorization = metadata.get("Authorization"); if (authorization == null || authorization.isEmpty() diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index b9e4d5d45e0f38..9d196237e34050 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -135,6 +135,9 @@ public static GcsFileSystemProperties of(Map properties) { @Override public void validate() { validateHmacDialect("gs", "GCS"); + if (auth.getNativeCredential().isPresent()) { + GcpAuth.validateEndpoint(getEndpoint()); + } } @Override diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java index abc45ba80c8372..a8ffa5f2df75c7 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java @@ -89,4 +89,20 @@ public void testBearerTokenInterceptorRejectsMissingAuthorizationMetadata() thro () -> interceptor.modifyHttpRequest(context, new ExecutionAttributes())); Assertions.assertTrue(exception.getMessage().contains("Failed to obtain a GCP access token")); } + + @Test + public void testUntrustedResolvedRequestIsRejectedBeforeFetchingToken() { + GoogleCredentials credentials = Mockito.mock(GoogleCredentials.class); + ExecutionInterceptor interceptor = GcpAuth.bearerTokenInterceptor(credentials); + Context.ModifyHttpRequest context = Mockito.mock(Context.ModifyHttpRequest.class); + for (String endpoint : new String[] {"https://attacker.example/bucket/key", + "http://storage.googleapis.com/bucket/key", "https://storage.googleapis.com.attacker.example/key", + "https://storage.googleapis.com:8443/key"}) { + Mockito.when(context.httpRequest()).thenReturn(SdkHttpRequest.builder() + .uri(URI.create(endpoint)).method(SdkHttpMethod.GET).build()); + Assertions.assertThrows(IllegalArgumentException.class, + () -> interceptor.modifyHttpRequest(context, new ExecutionAttributes())); + } + Mockito.verifyNoInteractions(credentials); + } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index 684358a1d57305..b77119c384004b 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -116,8 +116,8 @@ void toHadoopConfigurationMap_usesNativeAdcWithoutStaticCredentials() { @Test void nativeHadoopUsesConfiguredEndpoint() { - for (String endpoint : new String[] {"private-gcs.example.test", "https://private-gcs.example.test", - "https://private-gcs.example.test/", "http://localhost:4443"}) { + for (String endpoint : new String[] {"storage.googleapis.com", "https://storage.googleapis.com/", + "https://storage.us-central1.rep.googleapis.com", "https://us-central1-storage.googleapis.com"}) { GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( "gs.endpoint", endpoint, "gs.credential_provider_type", "COMPUTE_ENGINE")); String expected = endpoint.contains("://") ? endpoint : "https://" + endpoint; @@ -126,6 +126,20 @@ void nativeHadoopUsesConfiguredEndpoint() { } } + @Test + void nativeOauthRejectsUntrustedEndpointsButPreservesHmacAndAnonymous() { + for (String endpoint : new String[] {"https://attacker.example", "http://storage.googleapis.com", + "https://storage.googleapis.com.attacker.example", "https://storage.googleapis.com@attacker.example", + "https://storage.googleapis.com:8443"}) { + Assertions.assertThrows(IllegalArgumentException.class, + () -> GcsFileSystemProperties.of(Map.of("gs.endpoint", endpoint))); + Assertions.assertDoesNotThrow(() -> GcsFileSystemProperties.of(Map.of( + "gs.endpoint", endpoint, "gs.access_key", "ak", "gs.secret_key", "sk"))); + Assertions.assertDoesNotThrow(() -> GcsFileSystemProperties.of(Map.of( + "gs.endpoint", endpoint, "gs.credential_provider_type", "ANONYMOUS"))); + } + } + @Test void toHadoopConfigurationMap_simpleProviderWithStaticCredentials() { Map cfg = GcsFileSystemProperties.of(Map.of( From a3ce68cababe9a0814fcd6925c27a0cb3ad9310c Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 11:03:55 +0800 Subject: [PATCH 07/47] [fix](build) Fix GCP SDK error construction and FE test mocks ### What problem does this PR solve? Related PR: #68540 Problem Summary: The pinned AWS SDK does not expose the four-argument constructor on S3Error, so GCP authentication failures prevent BE compilation. Construct AWSError first and convert it to S3Error, preserving the error details and response codes. FE test compilation also fails because new S3 resource and vault tests use removed JMockit APIs. Migrate those mocks to scoped Mockito instances and retain the credential and no-RPC assertions. ### Release note None ### Check List (For Author) - Test: FE-core Checkstyle, clang-format 16, git diff --check, and PR-wide legacy JMockit scan passed. Constructor APIs verified against AWS SDK 1.11.221 headers. Full build and unit tests unavailable locally because JDK 17 and thirdparty dependencies are missing. - Behavior changed: No - Does this need documentation: No --- .../cpp/obj-client/auth/gcp/gcp_s3_client.cpp | 18 ++-- .../apache/doris/catalog/S3ResourceTest.java | 54 ++++++------ .../cloud/catalog/S3StorageVaultTest.java | 88 +++++++++---------- 3 files changed, 80 insertions(+), 80 deletions(-) diff --git a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp index 8b40e666453562..fe553cc1ada851 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_s3_client.cpp @@ -85,21 +85,23 @@ std::optional GcpS3Client::authorize_request( return (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '-' || c == '_' || c == '.'; }); if (!_trusted_endpoint || !valid_bucket) { - Aws::S3::S3Error error(Aws::S3::S3Errors::ACCESS_DENIED, "GcpEndpointError", - "Native GCP OAuth requires a trusted Google Cloud Storage HTTPS " - "endpoint on port 443 and a GCS bucket name", - false); + Aws::Client::AWSError error( + Aws::S3::S3Errors::ACCESS_DENIED, "GcpEndpointError", + "Native GCP OAuth requires a trusted Google Cloud Storage HTTPS " + "endpoint on port 443 and a GCS bucket name", + false); error.SetResponseCode(Aws::Http::HttpResponseCode::FORBIDDEN); - return error; + return Aws::S3::S3Error(std::move(error)); } auto token = fetch_token(); if (!token.has_value() || token->empty()) { // Return through the SDK outcome path before any object-storage HTTP request. // The token provider has already logged the underlying credential error. - Aws::S3::S3Error error(Aws::S3::S3Errors::ACCESS_DENIED, "GcpAuthenticationError", - "Failed to obtain GCP access token", false); + Aws::Client::AWSError error(Aws::S3::S3Errors::ACCESS_DENIED, + "GcpAuthenticationError", + "Failed to obtain GCP access token", false); error.SetResponseCode(Aws::Http::HttpResponseCode::UNAUTHORIZED); - return error; + return Aws::S3::S3Error(std::move(error)); } request.SetAdditionalCustomHeaderValue("Authorization", Aws::String("Bearer ") + token->c_str()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 1e83f7cf712d59..5ae73b8adb1ba4 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -430,31 +430,35 @@ public void testGsEndpointAndAlterAliases() throws Exception { } @Test - public void testGsEndpointForStorageVault(@Mocked Env env) throws Exception { - for (String provider : new String[] {"GCP", "gcp", "S3", "OSS"}) { - Map properties = new HashMap<>(s3Properties); - properties.put("provider", provider); - properties.put("gs.endpoint", "https://storage.googleapis.com"); - properties.put("gs.access_key", "gcp-access"); - properties.put("gs.secret_key", "gcp-secret"); - properties.put("gs.use_path_style", "true"); - properties.put("s3.root.path", "prefix"); - CreateResourceCommand command = new CreateResourceCommand( - new CreateResourceInfo(false, false, "gcp_vault", ImmutableMap.copyOf(properties))); - command.getInfo().analyzeResourceType(); - S3StorageVault vault = new S3StorageVault("gcp_vault", false, false, command); - vault.checkCreationProperties(ImmutableMap.copyOf(properties)); - Assertions.assertEquals("prefix", vault.getCopiedProperties().get(S3ResourceCompat.ROOT_PATH)); - String expectedEndpoint = "GCP".equalsIgnoreCase(provider) - ? "https://storage.googleapis.com" : s3Endpoint; - Assertions.assertEquals(expectedEndpoint, - CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getEndpoint()); - Assertions.assertEquals("GCP".equalsIgnoreCase(provider) ? "gcp-access" : s3AccessKey, - CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getAk()); - Assertions.assertEquals("GCP".equalsIgnoreCase(provider) ? "gcp-secret" : s3SecretKey, - CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getSk()); - Assertions.assertEquals("GCP".equalsIgnoreCase(provider), - CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getUsePathStyle()); + public void testGsEndpointForStorageVault() throws Exception { + Env env = Mockito.mock(Env.class); + try (MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + for (String provider : new String[] {"GCP", "gcp", "S3", "OSS"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", provider); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put("gs.access_key", "gcp-access"); + properties.put("gs.secret_key", "gcp-secret"); + properties.put("gs.use_path_style", "true"); + properties.put("s3.root.path", "prefix"); + CreateResourceCommand command = new CreateResourceCommand( + new CreateResourceInfo(false, false, "gcp_vault", ImmutableMap.copyOf(properties))); + command.getInfo().analyzeResourceType(); + S3StorageVault vault = new S3StorageVault("gcp_vault", false, false, command); + vault.checkCreationProperties(ImmutableMap.copyOf(properties)); + Assertions.assertEquals("prefix", vault.getCopiedProperties().get(S3ResourceCompat.ROOT_PATH)); + String expectedEndpoint = "GCP".equalsIgnoreCase(provider) + ? "https://storage.googleapis.com" : s3Endpoint; + Assertions.assertEquals(expectedEndpoint, + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getEndpoint()); + Assertions.assertEquals("GCP".equalsIgnoreCase(provider) ? "gcp-access" : s3AccessKey, + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getAk()); + Assertions.assertEquals("GCP".equalsIgnoreCase(provider) ? "gcp-secret" : s3SecretKey, + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getSk()); + Assertions.assertEquals("GCP".equalsIgnoreCase(provider), + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getUsePathStyle()); + } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java index 9b8d168de4d14b..619eab10f80080 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java @@ -23,18 +23,16 @@ import org.apache.doris.catalog.StorageVaultMgr; import org.apache.doris.cloud.proto.Cloud; import org.apache.doris.cloud.rpc.MetaServiceProxy; -import org.apache.doris.common.Config; import org.apache.doris.common.DdlException; import org.apache.doris.datasource.storage.S3ResourceCompat; import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.system.SystemInfoService; -import mockit.Mock; -import mockit.MockUp; -import mockit.Mocked; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; import java.util.HashMap; import java.util.Map; @@ -44,26 +42,23 @@ public class S3StorageVaultTest { private static final String ROLE = "arn:aws:iam::123456789012:role/test-role"; private static final String ACCOUNT = "target@my-project.iam.gserviceaccount.com"; - @Mocked - private Env env; - // Capture the actual CREATE/ALTER RPC, then stop before cache updates and backend synchronization. private Cloud.ObjectStoreInfoPB captureRequest(Map auth, boolean isAlter) throws Exception { AtomicReference captured = new AtomicReference<>(); - new MockUp() { - @Mock - public Cloud.AlterObjStoreInfoResponse alterStorageVault(Cloud.AlterObjStoreInfoRequest request) { - captured.set(request); - return Cloud.AlterObjStoreInfoResponse.newBuilder().setStatus( - Cloud.MetaServiceResponseStatus.newBuilder().setCode(Cloud.MetaServiceCode.INVALID_ARGUMENT) - .setMsg("captured vault request")).build(); - } - }; - String oldEndpoint = Config.meta_service_endpoint; - String oldCloudId = Config.cloud_unique_id; - Config.meta_service_endpoint = "127.0.0.1:20121"; - Config.cloud_unique_id = "vault-default-test"; - try { + Env env = Mockito.mock(Env.class); + MetaServiceProxy proxy = Mockito.mock(MetaServiceProxy.class); + try (MockedStatic mockedProxy = Mockito.mockStatic(MetaServiceProxy.class); + MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + mockedProxy.when(MetaServiceProxy::getInstance).thenReturn(proxy); + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + Mockito.when(proxy.alterStorageVault(Mockito.any(Cloud.AlterObjStoreInfoRequest.class))) + .thenAnswer(invocation -> { + captured.set(invocation.getArgument(0)); + return Cloud.AlterObjStoreInfoResponse.newBuilder().setStatus( + Cloud.MetaServiceResponseStatus.newBuilder() + .setCode(Cloud.MetaServiceCode.INVALID_ARGUMENT) + .setMsg("captured vault request")).build(); + }); StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); if (isAlter) { Assertions.assertThrows(DdlException.class, @@ -92,9 +87,6 @@ public Cloud.AlterObjStoreInfoResponse alterStorageVault(Cloud.AlterObjStoreInfo "ALTER must not send an inferred immutable provider"); } return captured.get().getVault().getObjInfo(); - } finally { - Config.meta_service_endpoint = oldEndpoint; - Config.cloud_unique_id = oldCloudId; } } @@ -209,30 +201,32 @@ public void testGcpCreateWithoutAuthenticationDefaultsToAdc() throws Exception { @Test public void testGcpAnonymousAliasesAreRejectedBeforeCreateRpc() throws Exception { - new MockUp() { - @Mock - public Cloud.AlterObjStoreInfoResponse alterStorageVault(Cloud.AlterObjStoreInfoRequest request) { - throw new AssertionError("Anonymous GCS vaults must be rejected before the MS RPC"); + Env env = Mockito.mock(Env.class); + MetaServiceProxy proxy = Mockito.mock(MetaServiceProxy.class); + try (MockedStatic mockedProxy = Mockito.mockStatic(MetaServiceProxy.class); + MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + mockedProxy.when(MetaServiceProxy::getInstance).thenReturn(proxy); + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); + for (String key : new String[] {GcpCredential.CREDENTIAL_PROVIDER_TYPE, + S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE}) { + Map properties = new HashMap<>(); + properties.put("type", "S3"); + properties.put("provider", "GCP"); + properties.put(S3ResourceCompat.ENDPOINT, "storage.googleapis.com"); + properties.put(S3ResourceCompat.REGION, "us-east1"); + properties.put(S3ResourceCompat.BUCKET, "test-bucket"); + properties.put(S3ResourceCompat.ROOT_PATH, "test-root"); + properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); + properties.put(key, " anonymous "); + CreateStorageVaultCommand command = new CreateStorageVaultCommand(false, "anonymous_gcp", properties); + command.setStorageVaultType(StorageVaultType.S3); + StorageVault vault = StorageVault.fromCommand(command); + IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, + () -> mgr.createS3Vault(vault), key); + Assertions.assertTrue(error.getMessage().contains("storage vaults"), key); } - }; - StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); - for (String key : new String[] {GcpCredential.CREDENTIAL_PROVIDER_TYPE, - S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE}) { - Map properties = new HashMap<>(); - properties.put("type", "S3"); - properties.put("provider", "GCP"); - properties.put(S3ResourceCompat.ENDPOINT, "storage.googleapis.com"); - properties.put(S3ResourceCompat.REGION, "us-east1"); - properties.put(S3ResourceCompat.BUCKET, "test-bucket"); - properties.put(S3ResourceCompat.ROOT_PATH, "test-root"); - properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); - properties.put(key, " anonymous "); - CreateStorageVaultCommand command = new CreateStorageVaultCommand(false, "anonymous_gcp", properties); - command.setStorageVaultType(StorageVaultType.S3); - StorageVault vault = StorageVault.fromCommand(command); - IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, - () -> mgr.createS3Vault(vault), key); - Assertions.assertTrue(error.getMessage().contains("storage vaults"), key); + Mockito.verifyNoInteractions(proxy); } } From 7dc131d9cdb2b96153e9e8a21af258afe7922704 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 14:33:57 +0800 Subject: [PATCH 08/47] [fix](storage) Correct native GCP identity and signed URL handling ### What problem does this PR solve? Related PR: #68540 Problem Summary: Legacy object-store ADD rejects a changed GCP identity because duplicate checks compare only the empty static keys. Include the native credential envelope in both duplicate scans. Native endpoint validation also accepts bucket-qualified hosts that the S3 resolvers prefix again; require service-base overrides while allowing resolved bucket hosts in the FE interceptor. Error-log URLs now avoid an extra key separator for empty prefixes and omit the default HTTPS port from both the signed Host and emitted URL. ### Release note Fix native GCP identity changes and error-log download URLs. Reject bucket-qualified native GCP endpoint overrides; configure a service-base endpoint and bucket separately. ### Check List (For Author) - Test: Added Cloud identity/duplicate tests, FE/BE endpoint validation tests, filesystem-to-signer key tests, and signed URL canonicalization tests. Checkstyle, clang-format 16, build hygiene and git diff --check passed. Compiled current C++ signer/validator in an isolated smoke probe, exercised extracted FE validation methods, and verified the generated canonical request hash against a real curl request to a local HTTPS listener. Full Doris unit tests remain unavailable locally due to missing JDK 17 and thirdparty dependencies; no live GCP retrieval performed. - Behavior changed: Yes, native GCP identity and signed URL handling corrected as described above. - Does this need documentation: No new properties; endpoint requirements are explained by validation errors. --- be/src/io/fs/s3_file_system.cpp | 4 +- .../common/auth/gcp/gcs_signed_url_test.cpp | 68 +++++++++++++++++++ be/test/io/s3_client_factory_test.cpp | 54 +++++++++++++-- .../meta-service/meta_service_resource.cpp | 11 ++- cloud/test/meta_service_test.cpp | 27 ++++++++ common/cpp/obj-client/auth/gcp/gcp_auth.cpp | 4 +- .../obj-client/auth/gcp/gcs_signed_url.cpp | 15 ++-- .../apache/doris/filesystem/gcs/GcpAuth.java | 14 ++-- .../doris/filesystem/gcs/GcpAuthTest.java | 23 +++++++ .../gcs/GcsFileSystemPropertiesTest.java | 4 +- 10 files changed, 201 insertions(+), 23 deletions(-) diff --git a/be/src/io/fs/s3_file_system.cpp b/be/src/io/fs/s3_file_system.cpp index 4797b573961a77..75298d6a65e3a1 100644 --- a/be/src/io/fs/s3_file_system.cpp +++ b/be/src/io/fs/s3_file_system.cpp @@ -445,7 +445,9 @@ Status S3FileSystem::download_impl(const Path& remote_file, const Path& local_fi // whether to return a public endpoint. std::string S3FileSystem::generate_presigned_url(const Path& path, int64_t expiration_secs, bool is_public_endpoint) const { - std::string key = fmt::format("{}/{}", _prefix, path.native()); + // An empty prefix must not add a separator absent from the uploaded object key. + std::string key = + _prefix.empty() ? path.native() : fmt::format("{}/{}", _prefix, path.native()); std::shared_ptr client; if (is_public_endpoint && _client->s3_client_conf().endpoint.ends_with(OSS_PRIVATE_ENDPOINT_SUFFIX)) { diff --git a/be/test/common/auth/gcp/gcs_signed_url_test.cpp b/be/test/common/auth/gcp/gcs_signed_url_test.cpp index f4a6292ae23b63..4c86e60972616e 100644 --- a/be/test/common/auth/gcp/gcs_signed_url_test.cpp +++ b/be/test/common/auth/gcp/gcs_signed_url_test.cpp @@ -55,6 +55,74 @@ TEST(GcsSignedUrlTest, BuildsV4UrlAndPassesCanonicalStringToSigner) { "f741db2e699530357ae34622ae689c7ec43e10e418af0ff7f1487beb1f144ce5"); } +TEST(GcsSignedUrlTest, DefaultHttpsPortProducesTheSameSignatureAndUrl) { + GcsV4SignedUrlOptions options { + .endpoint = "https://storage.googleapis.com", + .bucket = "test-bucket", + .key = "error_log/id", + .signer_email = "signer@my-project.iam.gserviceaccount.com", + .expiration_secs = 60, + }; + std::string without_port; + auto expected = build_gcs_v4_signed_url(options, std::chrono::system_clock::time_point {}, + [&](std::string_view value) { + without_port = value; + return GcsSignBlobResult {.signature = "test"}; + }); + ASSERT_TRUE(expected.ok()) << expected.error; + for (const auto* endpoint : + {"https://storage.googleapis.com:443", "storage.googleapis.com:443/"}) { + options.endpoint = endpoint; + std::string with_port; + auto result = build_gcs_v4_signed_url(options, std::chrono::system_clock::time_point {}, + [&](std::string_view value) { + with_port = value; + return GcsSignBlobResult {.signature = "test"}; + }); + ASSERT_TRUE(result.ok()) << result.error; + EXPECT_EQ(result.signed_url, expected.signed_url); + EXPECT_EQ(with_port, without_port); + } +} + +TEST(GcsSignedUrlTest, RejectsBucketEndpointsBeforeSigning) { + GcsV4SignedUrlOptions options { + .bucket = "test-bucket", + .key = "error_log/id", + .signer_email = "signer@my-project.iam.gserviceaccount.com", + .expiration_secs = 60, + }; + for (const auto* endpoint : {"https://test-bucket.storage.googleapis.com", + "https://test-bucket.us-central1-storage.googleapis.com", + "https://test-bucket.storage.us-central1.rep.googleapis.com"}) { + options.endpoint = endpoint; + bool signer_called = false; + auto result = build_gcs_v4_signed_url(options, std::chrono::system_clock::time_point {}, + [&](std::string_view) { + signer_called = true; + return GcsSignBlobResult {.signature = "test"}; + }); + EXPECT_FALSE(result.ok()) << endpoint; + EXPECT_FALSE(signer_called); + } +} + +TEST(GcsSignedUrlTest, PreservesLiteralLeadingSlashInObjectKey) { + GcsV4SignedUrlOptions options { + .endpoint = "https://storage.googleapis.com", + .bucket = "test-bucket", + .key = "/error_log/id", + .signer_email = "signer@my-project.iam.gserviceaccount.com", + .expiration_secs = 60, + }; + auto result = build_gcs_v4_signed_url( + options, std::chrono::system_clock::time_point {}, + [](std::string_view) { return GcsSignBlobResult {.signature = "test"}; }); + ASSERT_TRUE(result.ok()) << result.error; + EXPECT_TRUE(result.signed_url.starts_with( + "https://storage.googleapis.com/test-bucket//error_log/id?")); +} + TEST(GcsSignedUrlTest, RejectsInvalidExpirationBeforeSigning) { GcsV4SignedUrlOptions options { .endpoint = "storage.googleapis.com", diff --git a/be/test/io/s3_client_factory_test.cpp b/be/test/io/s3_client_factory_test.cpp index 097efcb7665d30..63524af7a44690 100644 --- a/be/test/io/s3_client_factory_test.cpp +++ b/be/test/io/s3_client_factory_test.cpp @@ -50,6 +50,7 @@ #include #include +#include #include #include #include @@ -68,6 +69,7 @@ #include "cpp/obj-client/auth/aws_credential_factory.h" #include "cpp/obj-client/auth/gcp/gcp_auth.h" #include "cpp/obj-client/auth/gcp/gcp_s3_client.h" +#include "cpp/obj-client/auth/gcp/gcs_signed_url.h" #include "cpp/obj-client/rate_limited_obj_storage_client.h" #include "cpp/obj-client/s3_obj_storage_client.h" #include "cpp/sync_point.h" @@ -432,10 +434,13 @@ TEST_F(S3ClientFactoryTest, GcpBearerHeaderSurvivesAnonymousAwsSigning) { } TEST_F(S3ClientFactoryTest, GcpRejectsUntrustedEndpointsBeforeFetchingToken) { - for (const auto* endpoint : {"https://attacker.example", "http://storage.googleapis.com", - "https://storage.googleapis.com.attacker.example", - "https://storage.googleapis.com@attacker.example", - "https://storage.googleapis.com:8443", ""}) { + for (const auto* endpoint : + {"https://attacker.example", "http://storage.googleapis.com", + "https://storage.googleapis.com.attacker.example", + "https://storage.googleapis.com@attacker.example", "https://storage.googleapis.com:8443", + "https://bucket.storage.googleapis.com", + "https://bucket.us-central1-storage.googleapis.com", + "https://bucket.storage.us-central1.rep.googleapis.com", ""}) { FixedTokenGcpS3Client client(endpoint); Aws::S3::Model::GetObjectRequest request; request.SetBucket("bucket"); @@ -454,13 +459,52 @@ TEST_F(S3ClientFactoryTest, GcpRejectsUntrustedEndpointsBeforeFetchingToken) { TEST_F(S3ClientFactoryTest, GcpAcceptsGoogleStorageEndpoints) { for (const auto* endpoint : - {"https://storage.googleapis.com", "https://bucket.storage.googleapis.com:443/", + {"https://storage.googleapis.com", "https://storage.googleapis.com:443/", "https://us-central1-storage.googleapis.com", "https://storage.us-central1.rep.googleapis.com"}) { EXPECT_TRUE(is_valid_gcp_storage_endpoint(endpoint)) << endpoint; } } +TEST_F(S3ClientFactoryTest, GcpErrorLogPresigningUsesUploadedKey) { + for (const std::string prefix : {"", "instance-prefix"}) { + S3Conf conf; + conf.bucket = "test-bucket"; + conf.prefix = prefix; + io::S3FileSystem fs(std::move(conf), "gcp-error-log-test"); + const std::string expected_key = + prefix.empty() ? "error_log/id" : "instance-prefix/error_log/id"; + io::Path upload_path; + ASSERT_TRUE(fs.absolute_path("error_log/id", upload_path).ok()); + S3URI upload_uri(upload_path.native()); + ASSERT_TRUE(upload_uri.parse().ok()); + ASSERT_EQ(upload_uri.get_key(), expected_key); + + fs.client_holder()->_client = std::make_shared( + nullptr, ObjStorageEndpointInfo {}, + [&](const ObjStoragePath& path, int64_t expiration_secs) { + EXPECT_EQ(path.key, upload_uri.get_key()); + GcsV4SignedUrlOptions options { + .endpoint = "https://storage.googleapis.com", + .bucket = path.bucket, + .key = path.key, + .signer_email = "signer@test.iam.gserviceaccount.com", + .expiration_secs = expiration_secs, + }; + auto result = build_gcs_v4_signed_url( + options, std::chrono::system_clock::time_point {}, + [](std::string_view) { + return GcsSignBlobResult {.signature = "test"}; + }); + EXPECT_TRUE(result.ok()) << result.error; + return result.signed_url; + }); + EXPECT_TRUE(fs.generate_presigned_url("error_log/id", 60, false) + .starts_with("https://storage.googleapis.com/test-bucket/" + + expected_key + "?")); + } +} + TEST_F(S3ClientFactoryTest, GcpTokenFailureRejectsEveryObjectStorageOperation) { FixedTokenGcpS3Client client; for (const auto& token : {std::optional {}, std::optional {""}}) { diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index a78dd505ecb23b..8c287468683087 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -1609,7 +1610,10 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr for (auto& it : objs) { if (bucket == it.bucket() && prefix == it.prefix() && endpoint == it.endpoint() && region == it.region() && ak == it.ak() && sk == it.sk() && - obj.provider() == it.provider() && external_endpoint == it.external_endpoint()) { + obj.provider() == it.provider() && external_endpoint == it.external_endpoint() && + obj.has_credential() == it.has_credential() && + google::protobuf::util::MessageDifferencer::Equivalent(obj.credential(), + it.credential())) { // err, anything not changed code = MetaServiceCode::INVALID_ARGUMENT; msg = "original obj infos has a same conf, please check it"; @@ -2012,7 +2016,10 @@ void MetaServiceImpl::alter_obj_store_info(google::protobuf::RpcController* cont for (auto& it : objs) { if (bucket == it.bucket() && prefix == it.prefix() && endpoint == it.endpoint() && region == it.region() && ak == it.ak() && sk == it.sk() && - obj.provider() == it.provider() && external_endpoint == it.external_endpoint()) { + obj.provider() == it.provider() && external_endpoint == it.external_endpoint() && + obj.has_credential() == it.has_credential() && + google::protobuf::util::MessageDifferencer::Equivalent(obj.credential(), + it.credential())) { // err, anything not changed code = MetaServiceCode::INVALID_ARGUMENT; msg = "original obj infos has a same conf, please check it"; diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index d28ca5d62310ea..1b2bf04b88c917 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -5074,6 +5074,33 @@ TEST(MetaServiceTest, NativeGcpLegacyAddReadback) { obj->credential().SerializeAsString()); ASSERT_FALSE(instance.obj_info(0).has_ak()); ASSERT_FALSE(instance.obj_info(0).has_encryption_info()); + + // A changed account or credential source at the same location is a new configuration. + const auto original_credential = instance.obj_info(0).credential().SerializeAsString(); + int expected_count = 1; + credential->set_impersonation_service_account("rotated@test.iam.gserviceaccount.com"); + for (auto provider : {GcpCredentialPB::COMPUTE_ENGINE, GcpCredentialPB::DEFAULT}) { + credential->set_credential_provider_type(provider); + res.Clear(); + meta_service->alter_obj_store_info(&cntl, &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + ++expected_count; + + // An exact repeat must still be rejected without appending another entry. + res.Clear(); + meta_service->alter_obj_store_info(&cntl, &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT); + EXPECT_EQ(res.status().msg(), "original obj infos has a same conf, please check it"); + + txn.reset(); + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + ASSERT_EQ(txn->get(key, &val), TxnErrorCode::TXN_OK); + ASSERT_TRUE(instance.ParseFromString(val)); + ASSERT_EQ(instance.obj_info_size(), expected_count); + EXPECT_EQ(instance.obj_info(0).credential().SerializeAsString(), original_credential); + EXPECT_EQ(instance.obj_info(expected_count - 1).credential().SerializeAsString(), + obj->credential().SerializeAsString()); + } } TEST(MetaServiceTest, AddRoleOnlyVaultDefaultsToInstanceProfile) { diff --git a/common/cpp/obj-client/auth/gcp/gcp_auth.cpp b/common/cpp/obj-client/auth/gcp/gcp_auth.cpp index f0af88056071c5..8928e202a34845 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_auth.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_auth.cpp @@ -25,9 +25,9 @@ namespace doris { bool is_valid_gcp_storage_endpoint(std::string_view endpoint) { - // Match the FE GcpAuth policy: Google-owned XML API hosts, HTTPS and port 443 only. + // Only service-base XML API endpoints: the S3 clients add the bucket themselves. static const std::regex pattern( - R"(https://([a-z0-9-]+\.)*(storage\.googleapis\.com|[a-z0-9-]+-storage\.googleapis\.com|storage\.[a-z0-9-]+\.rep\.googleapis\.com)(:443)?/?)", + R"(https://(storage\.googleapis\.com|[a-z0-9-]+-storage\.googleapis\.com|storage\.[a-z0-9-]+\.rep\.googleapis\.com)(:443)?/?)", std::regex::icase); return std::regex_match(endpoint.begin(), endpoint.end(), pattern); } diff --git a/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp b/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp index 9adbd09818ccf8..3a5e18a953965c 100644 --- a/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp +++ b/common/cpp/obj-client/auth/gcp/gcs_signed_url.cpp @@ -93,17 +93,16 @@ std::optional parse_endpoint(std::string endpoint, Endpoint* parsed parsed->scheme = endpoint.substr(0, scheme_end); endpoint.erase(0, scheme_end + 3); } - if (parsed->scheme != "http" && parsed->scheme != "https") { - return fmt::format("unsupported GCS endpoint scheme: {}", parsed->scheme); + if (!is_valid_gcp_storage_endpoint(parsed->scheme + "://" + endpoint)) { + return "GCS signed URL requires a Google Storage HTTPS service-base endpoint without a " + "bucket"; } - while (endpoint.ends_with('/')) { + if (endpoint.ends_with('/')) { endpoint.pop_back(); } - if (endpoint.empty() || endpoint.find_first_of("/?#") != std::string::npos) { - return fmt::format( - "GCS signed URL endpoint must not contain a path, query, " - "or fragment: {}", - endpoint); + // HTTP clients omit the default TLS port from Host. Sign the same authority. + if (endpoint.ends_with(":443")) { + endpoint.resize(endpoint.size() - 4); } parsed->authority = std::move(endpoint); return std::nullopt; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java index 75cb284d6b0da1..dbcf265588966a 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java @@ -53,11 +53,13 @@ public final class GcpAuth { "https://www.googleapis.com/auth/devstorage.read_write"; private static final String CLOUD_PLATFORM_SCOPE = "https://www.googleapis.com/auth/cloud-platform"; private static final int IMPERSONATION_LIFETIME_SECONDS = 3600; - // Global, locational and regional XML API endpoints, including virtual-hosted buckets. + // Endpoint overrides must be service-base hosts; SDK resolvers add the bucket. + private static final Pattern STORAGE_BASE_HOST = Pattern.compile( + "(?:storage\\.googleapis\\.com|[a-z0-9-]+-storage\\.googleapis\\.com|" + + "storage\\.[a-z0-9-]+\\.rep\\.googleapis\\.com)", Pattern.CASE_INSENSITIVE); + // Resolved virtual-hosted requests legitimately contain the bucket prefix. private static final Pattern STORAGE_HOST = Pattern.compile( - "(?:[a-z0-9-]+\\.)*(?:storage\\.googleapis\\.com|" - + "[a-z0-9-]+-storage\\.googleapis\\.com|storage\\.[a-z0-9-]+\\.rep\\.googleapis\\.com)", - Pattern.CASE_INSENSITIVE); + "(?:[a-z0-9-]+\\.)*" + STORAGE_BASE_HOST.pattern(), Pattern.CASE_INSENSITIVE); private GcpAuth() { } @@ -91,6 +93,10 @@ private static URI endpoint(GcsFileSystemProperties properties) { static URI validateEndpoint(String endpoint) { URI uri = URI.create(endpoint.contains("://") ? endpoint : "https://" + endpoint); validateRequestUri(uri); + if (!STORAGE_BASE_HOST.matcher(uri.getHost()).matches()) { + throw new IllegalArgumentException( + "Native GCP OAuth endpoint must be a service-base host without a bucket"); + } if (uri.getRawQuery() != null || (!uri.getRawPath().isEmpty() && !"/".equals(uri.getRawPath()))) { throw new IllegalArgumentException("Native GCP OAuth endpoint must not contain a path or query"); } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java index a8ffa5f2df75c7..a79261c0d9b79f 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java @@ -90,6 +90,29 @@ public void testBearerTokenInterceptorRejectsMissingAuthorizationMetadata() thro Assertions.assertTrue(exception.getMessage().contains("Failed to obtain a GCP access token")); } + @Test + public void testEndpointOverridesRejectBucketHostsButResolvedRequestsAcceptThem() { + for (String endpoint : new String[] {"https://storage.googleapis.com", + "https://storage.googleapis.com:443/", "https://us-central1-storage.googleapis.com", + "https://storage.us-central1.rep.googleapis.com"}) { + Assertions.assertEquals(URI.create(endpoint), GcpAuth.validateEndpoint(endpoint)); + } + for (String endpoint : new String[] {"https://bucket.storage.googleapis.com", + "https://bucket.us-central1-storage.googleapis.com", + "https://bucket.storage.us-central1.rep.googleapis.com"}) { + Assertions.assertThrows(IllegalArgumentException.class, () -> GcpAuth.validateEndpoint(endpoint)); + // These hosts are still valid after the SDK adds the bucket to a service-base endpoint. + GoogleCredentials credentials = GoogleCredentials.create( + new AccessToken("metadata-token", new Date(System.currentTimeMillis() + 3_600_000))); + Context.ModifyHttpRequest context = Mockito.mock(Context.ModifyHttpRequest.class); + Mockito.when(context.httpRequest()).thenReturn(SdkHttpRequest.builder() + .uri(URI.create(endpoint + "/key")).method(SdkHttpMethod.GET).build()); + Assertions.assertEquals("Bearer metadata-token", GcpAuth.bearerTokenInterceptor(credentials) + .modifyHttpRequest(context, new ExecutionAttributes()) + .firstMatchingHeader("Authorization").orElse(null)); + } + } + @Test public void testUntrustedResolvedRequestIsRejectedBeforeFetchingToken() { GoogleCredentials credentials = Mockito.mock(GoogleCredentials.class); diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index b77119c384004b..8e444251e49ab3 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -130,7 +130,9 @@ void nativeHadoopUsesConfiguredEndpoint() { void nativeOauthRejectsUntrustedEndpointsButPreservesHmacAndAnonymous() { for (String endpoint : new String[] {"https://attacker.example", "http://storage.googleapis.com", "https://storage.googleapis.com.attacker.example", "https://storage.googleapis.com@attacker.example", - "https://storage.googleapis.com:8443"}) { + "https://storage.googleapis.com:8443", "https://bucket.storage.googleapis.com", + "https://bucket.us-central1-storage.googleapis.com", + "https://bucket.storage.us-central1.rep.googleapis.com"}) { Assertions.assertThrows(IllegalArgumentException.class, () -> GcsFileSystemProperties.of(Map.of("gs.endpoint", endpoint))); Assertions.assertDoesNotThrow(() -> GcsFileSystemProperties.of(Map.of( From 4c8c09699b0d9933cfccb304bf805eec18288e7f Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 14:48:19 +0800 Subject: [PATCH 09/47] [fix](fe) Fix native GCP storage routing and Iceberg test key ### What problem does this PR solve? Related PR: #68540 Problem Summary: Native GCP EXPORT properties without an explicit endpoint were claimed by S3's region fallback before the GCS provider could bind them, causing valid GCP credentials to be rejected. Yield to the existing GCS selectors before this fallback while preserving explicit S3 selection. The Iceberg native GCP catalog test also looked up an incorrect literal client-factory key; use Iceberg's S3FileIOProperties.CLIENT_FACTORY constant. ### Release note Fix native GCP storage selection when a region is set without an endpoint. ### Check List (For Author) - Test: Added provider routing regression cases; Checkstyle and connector import/metadata-funnel gates passed. An isolated probe using extracted production routing methods reproduces the old failure and passes 10 routing/auth cases after the fix; Iceberg 1.11.0 factory key verified. Full FE UT not run locally because JDK 17 and required protoc are absent. - Behavior changed: Yes, GCS selectors take priority over S3 region guessing. - Does this need documentation: No --- .../iceberg/IcebergCatalogFactoryTest.java | 4 ++- .../filesystem/s3/S3FileSystemProvider.java | 5 +++ .../s3/S3FileSystemProviderTest.java | 32 +++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index db6435d673783c..89d6563153379d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -30,6 +30,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.iceberg.aws.AwsClientProperties; +import org.apache.iceberg.aws.s3.S3FileIOProperties; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -1062,7 +1063,8 @@ void nativeGcpCatalogUsesBearerFileIoAndNativeHadoopConfiguration() { Map options = IcebergCatalogFactory.buildCatalogProperties( IcebergCatalogProperties.of(properties), Optional.of(storage)); Assertions.assertEquals("org.apache.iceberg.aws.s3.S3FileIO", options.get("io-impl")); - Assertions.assertEquals(GcpS3FileIOAwsClientFactory.class.getName(), options.get("s3.client-factory")); + Assertions.assertEquals(GcpS3FileIOAwsClientFactory.class.getName(), + options.get(S3FileIOProperties.CLIENT_FACTORY)); Assertions.assertEquals(source, options.get("gs.credential_provider_type")); Assertions.assertEquals("https://storage.googleapis.com", options.get("s3.endpoint")); Assertions.assertEquals("target@test.iam.gserviceaccount.com", diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java index a4f95a85fd0a66..18710ecbfad0ed 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java @@ -115,6 +115,11 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { + // GCS can use its default endpoint with only an S3-compatible region property. Yield + // before the region fallback so bindPrimary/bindAll select the GCS authentication path. + if (!S3CompatSignals.hasExplicitS3Request(properties) && S3CompatSignals.guessIsGcs(properties)) { + return false; + } // Port of fe-core S3Properties.guessIsMe on raw props. String endpoint = null; for (String name : GUESS_ENDPOINT_NAMES) { diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java index c81f119386bc97..2c3870384857d4 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java @@ -181,6 +181,38 @@ void supports_stillAcceptsUnmarkedGenericFallback() { Assertions.assertTrue(provider.supports(props)); } + @Test + void supportsGuess_yieldsToGcsWithoutAnExplicitEndpoint() { + for (Map.Entry hint : Map.of( + "provider", "GCP", + "gs.credential_provider_type", "DEFAULT", + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com", + "warehouse", "gs://bucket/warehouse").entrySet()) { + Map props = new HashMap<>(); + props.put("s3.region", "us-east1"); + props.put("_STORAGE_TYPE_", "S3"); + props.put(hint.getKey(), hint.getValue()); + + Assertions.assertFalse(provider.supportsGuess(props), "must yield for " + hint); + } + } + + @Test + void supportsGuess_preservesS3RegionFallbackAndExplicitSelection() { + Map props = new HashMap<>(); + props.put("s3.region", "us-east-1"); + Assertions.assertTrue(provider.supportsGuess(props)); + + props.put("provider", "S3"); + props.put("warehouse", "gs://bucket/warehouse"); + Assertions.assertTrue(provider.supportsGuess(props)); + + props.remove("provider"); + props.put("fs.s3.support", "true"); + Assertions.assertTrue(provider.supportsExplicit(props)); + Assertions.assertTrue(provider.supportsGuess(props)); + } + @Test void bind_returnsValidatedS3FileSystemProperties() { Map props = new HashMap<>(); From f65ef0e84f403bcb7dde123c6e0e116a0c022fb3 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 15:03:03 +0800 Subject: [PATCH 10/47] [fix](be) Initialize endpoint in GCS signed URL test ### What problem does this PR solve? Related PR: #68540 Problem Summary: The bucket-endpoint rejection test omitted endpoint from its designated initializer. BE UT compilation treats the resulting -Wmissing-designated-field-initializers warning as an error. Initialize the field to an empty string before the loop assigns each rejected endpoint. ### Release note None ### Check List (For Author) - Test: Extracted all five option initializers verbatim and compiled against the production header with C++20, -Wall -Wextra -Werror and -Wmissing-designated-field-initializers. Reproduced the failure before the fix; all initializers pass after it. clang-format 16 and git diff --check passed. Full BE UT not run locally due to missing build dependencies. - Behavior changed: No - Does this need documentation: No --- be/test/common/auth/gcp/gcs_signed_url_test.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/be/test/common/auth/gcp/gcs_signed_url_test.cpp b/be/test/common/auth/gcp/gcs_signed_url_test.cpp index 4c86e60972616e..5425fa7c979642 100644 --- a/be/test/common/auth/gcp/gcs_signed_url_test.cpp +++ b/be/test/common/auth/gcp/gcs_signed_url_test.cpp @@ -87,6 +87,7 @@ TEST(GcsSignedUrlTest, DefaultHttpsPortProducesTheSameSignatureAndUrl) { TEST(GcsSignedUrlTest, RejectsBucketEndpointsBeforeSigning) { GcsV4SignedUrlOptions options { + .endpoint = "", .bucket = "test-bucket", .key = "error_log/id", .signer_email = "signer@my-project.iam.gserviceaccount.com", From 888dee9a329db1a4f4086106ddf73f4c2168c95e Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 15:57:41 +0800 Subject: [PATCH 11/47] [fix](cloud) Clean up FilterCopyFilesTest sync callbacks ### What problem does this PR solve? Related PR: #68540 Problem Summary: FilterCopyFilesTest left SyncPoint enabled and retained a get_instance_id callback that captured its local instance_id by reference. The following native GCP test invoked that callback after the string had been destroyed, causing an ASAN stack-use-after-return failure. Use the existing scope guard to disable processing and clear callbacks before local variables are destroyed, including when an assertion returns early. ### Release note None ### Check List (For Author) - Test: An isolated probe using the actual SyncPoint implementation and callback setup reproduces cross-test contamination before the fix. With the fix, normal and early returns pass ASAN checks and leave all three callbacks removed and processing disabled. clang-format 16 and git diff --check passed. Full Cloud UT not run locally because required build dependencies are unavailable. - Behavior changed: No production behavior change; fixes test isolation. - Does this need documentation: No --- cloud/test/meta_service_test.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index 1b2bf04b88c917..5c8bdb7c91fab0 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -4759,6 +4759,10 @@ TEST(MetaServiceTest, FilterCopyFilesTest) { auto stage_id = "test_stage_id"; int64_t table_id = 100; [[maybe_unused]] auto sp = SyncPoint::get_instance(); + DORIS_CLOUD_DEFER { + sp->disable_processing(); + sp->clear_all_call_backs(); + }; sp->set_call_back("get_instance_id", [&](auto&& args) { auto* ret = try_any_cast_ret(args); ret->first = instance_id; From 4a5ae6843ffbc715364894612f412c7b5e75ea65 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 8 Oct 2026 17:38:03 +0800 Subject: [PATCH 12/47] [fix](auth) Protect native GCP endpoints and vault credentials ### What problem does this PR solve? Related PR: #68540 Problem Summary: Connector Hadoop configuration overlays raw catalog fs.* properties after the canonical GCS configuration. Native GCP property binding now rejects fs.gs.storage.root.url and fs.gs.storage.service.path so OAuth requests keep using the destination validated through gs.endpoint. ALTER STORAGE VAULT previously accepted empty replacement access/secret keys and removed a working native credential. Reject empty or half-empty replacement values before encryption or credential removal. Cover failure atomicity for both the persisted vault and a requested rename. ### Release note Native GCP Hadoop endpoints must be configured with gs.endpoint. Empty static-key replacements no longer erase native storage vault credentials. ### Check List (For Author) - Test: Added native Hadoop override rejection and HMAC compatibility cases, plus Cloud RPC/persistence cases for empty and half-empty key replacements. FE Checkstyle, clang-format 16 and git diff --check passed. Full FE/Cloud unit tests were not run locally because required build dependencies are unavailable; CI execution is required. - Behavior changed: Yes, reject raw Hadoop GCS endpoint overrides in native mode and preserve native vault auth on invalid static-key replacements. - Does this need documentation: No --- .../meta-service/meta_service_resource.cpp | 6 +++++ cloud/test/meta_service_test.cpp | 23 +++++++++++++++++++ .../gcs/GcsFileSystemProperties.java | 8 +++++++ .../gcs/GcsFileSystemPropertiesTest.java | 19 +++++++++++++++ 4 files changed, 56 insertions(+) diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index 8c287468683087..792d05fb90c30e 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -1186,6 +1186,12 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt } if (obj_info.has_ak()) { + if (has_obj_credential(new_vault.obj_info()) && + (obj_info.ak().empty() || obj_info.sk().empty())) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = "Replacing native credentials requires nonempty accesskey and secretkey"; + return -1; + } EncryptionInfoPB encryption_info = new_vault.obj_info().encryption_info(); AkSkPair new_ak_sk_pair {new_vault.obj_info().ak(), new_vault.obj_info().sk()}; diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index 5c8bdb7c91fab0..fb0327d2e8d880 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -956,6 +956,29 @@ TEST(MetaServiceTest, AlterS3StorageVaultTest) { ASSERT_EQ(stored_instance.storage_vault_names(1), gcp_vault_name); req.mutable_vault()->clear_alter_name(); + // Empty or half-empty key replacements must not remove native authentication + // or persist the rename that was applied to the transaction's candidate. + for (int empty_keys = 1; empty_keys <= 3; ++empty_keys) { + AlterObjStoreInfoRequest empty_req; + empty_req.set_cloud_unique_id("test_cloud_unique_id"); + empty_req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + empty_req.mutable_vault()->set_name(gcp_vault_name); + empty_req.mutable_vault()->set_alter_name("empty_key_rename"); + auto* empty_update = empty_req.mutable_vault()->mutable_obj_info(); + empty_update->set_ak((empty_keys & 1) ? "" : "replacement-ak"); + empty_update->set_sk((empty_keys & 2) ? "" : "replacement-sk"); + brpc::Controller empty_cntl; + AlterObjStoreInfoResponse empty_res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&empty_cntl), &empty_req, + &empty_res, nullptr); + ASSERT_EQ(empty_res.status().code(), MetaServiceCode::INVALID_ARGUMENT); + get_test_vault("3", stored); + ASSERT_EQ(stored.SerializeAsString(), original); + get_test_instance(stored_instance); + ASSERT_EQ(stored_instance.storage_vault_names(1), gcp_vault_name); + } + // Switching from encrypted static keys creates a new native credential. update->Clear(); update->set_ak("gcp-ak"); diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index 9d196237e34050..bca2b020314767 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -137,6 +137,14 @@ public void validate() { validateHmacDialect("gs", "GCS"); if (auth.getNativeCredential().isPresent()) { GcpAuth.validateEndpoint(getEndpoint()); + // Connectors overlay raw Hadoop properties after this canonical configuration. + // Keep the OAuth destination on the validated gs.endpoint path. + for (String key : Set.of("fs.gs.storage.root.url", "fs.gs.storage.service.path")) { + if (rawProperties().containsKey(key)) { + throw new IllegalArgumentException("Native GCP authentication does not allow " + key + + "; configure the storage endpoint with gs.endpoint instead"); + } + } } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index 8e444251e49ab3..1dd4e096a457f0 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -243,6 +243,25 @@ void nativeModesPropagateToBackendHadoopAndCache() { } } + @Test + void nativeHadoopEndpointCannotBeOverriddenByRawProperties() { + for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + for (String key : new String[] {"fs.gs.storage.root.url", "fs.gs.storage.service.path"}) { + IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, + () -> GcsFileSystemProperties.of(Map.of("provider", "GCP", + "gs.credential_provider_type", mode, key, "https://untrusted.example/"))); + Assertions.assertTrue(error.getMessage().contains(key)); + } + } + GcsFileSystemProperties nativeProps = GcsFileSystemProperties.of(Map.of("provider", "GCP", + "gs.endpoint", "https://us-central1-storage.googleapis.com")); + Assertions.assertEquals("https://us-central1-storage.googleapis.com/", + nativeProps.toHadoopConfigurationMap().get("fs.gs.storage.root.url")); + // The restriction applies to native OAuth; preserve legacy HMAC Hadoop configuration. + Assertions.assertDoesNotThrow(() -> GcsFileSystemProperties.of(Map.of("provider", "GCP", + "gs.access_key", "ak", "gs.secret_key", "sk", "fs.gs.storage.root.url", "https://custom.example/"))); + } + @Test void explicitAnonymousUsesUnsignedBackendAndHadoop() { GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( From 60254be8f341aa81ea131796e2001568dc105f7a Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 10:57:56 +0800 Subject: [PATCH 13/47] [fix](fe) Reject raw Hadoop auth overrides for native GCP ### What problem does this PR solve? Related PR: #68540 Problem Summary: Native GCP storage selects one identity for object clients, but Hadoop catalog configuration overlays raw fs.* properties afterward. An fs.gs.auth.type=UNAUTHENTICATED override or a different impersonation account therefore makes Hadoop namespace operations use a different identity. Reject raw fs.gs.auth.* properties during native GCS binding and direct users to the canonical gs.credential_provider_type and gs.impersonation_service_account properties. Preserve HMAC, anonymous, and unrelated Hadoop configuration. ### Release note Native GCP authentication rejects raw fs.gs.auth.* overrides to keep Hadoop catalog and object operations on the selected GCP identity. ### Check List (For Author) - Test: Add unit coverage for implicit ADC, explicit native modes, impersonation replacement/clearing, and legacy-mode compatibility. GCS Checkstyle and git diff --check pass. Standard run-fe-ut.sh was attempted but stopped before tests because the installed JDK 17 fails to initialize. - Behavior changed: Yes; conflicting raw Hadoop auth configuration fails early. - Does this need documentation: Yes; native GCP properties use the canonical gs.* authentication settings. --- .../gcs/GcsFileSystemProperties.java | 8 +++ .../gcs/GcsFileSystemPropertiesTest.java | 51 +++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index bca2b020314767..5244fedd685af6 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -145,6 +145,14 @@ public void validate() { + "; configure the storage endpoint with gs.endpoint instead"); } } + // Raw auth overrides would make Hadoop use a different identity from object clients. + for (String key : rawProperties().keySet()) { + if (key.startsWith("fs.gs.auth.")) { + throw new IllegalArgumentException("Native GCP authentication does not allow " + key + + "; configure authentication with " + GcpCredential.CREDENTIAL_PROVIDER_TYPE + + " and " + GcpCredential.IMPERSONATION_SERVICE_ACCOUNT + " instead"); + } + } } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index 1dd4e096a457f0..5db5b350d71203 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -20,6 +20,8 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.HashMap; +import java.util.List; import java.util.Map; class GcsFileSystemPropertiesTest { @@ -262,6 +264,55 @@ void nativeHadoopEndpointCannotBeOverriddenByRawProperties() { "gs.access_key", "ak", "gs.secret_key", "sk", "fs.gs.storage.root.url", "https://custom.example/"))); } + @Test + void nativeHadoopAuthCannotBeOverriddenByRawProperties() { + List> nativeModes = List.of( + Map.of("provider", "GCP"), + Map.of("provider", "GCP", "gs.credential_provider_type", "DEFAULT"), + Map.of("provider", "GCP", "gs.credential_provider_type", "COMPUTE_ENGINE")); + Map overrides = Map.of( + "fs.gs.auth.type", "UNAUTHENTICATED", + "fs.gs.auth.impersonation.service.account", "other@test.iam.gserviceaccount.com", + "fs.gs.auth.service.account.json.keyfile", "/tmp/other-service-account.json"); + for (Map mode : nativeModes) { + for (String account : new String[] {"", "target@test.iam.gserviceaccount.com"}) { + overrides.forEach((key, value) -> { + Map properties = new HashMap<>(mode); + if (!account.isEmpty()) { + properties.put("gs.impersonation_service_account", account); + } + properties.put(key, value); + IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, + () -> GcsFileSystemProperties.of(properties)); + Assertions.assertTrue(error.getMessage().contains(key)); + Assertions.assertTrue(error.getMessage().contains("gs.credential_provider_type")); + Assertions.assertTrue(error.getMessage().contains("gs.impersonation_service_account")); + }); + } + } + // An empty raw value must not clear the selected impersonation identity either. + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com", + "fs.gs.auth.impersonation.service.account", ""))); + } + + @Test + void nativeHadoopAuthValidationPreservesOtherModesAndUnrelatedProperties() { + Assertions.assertDoesNotThrow(() -> GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.access_key", "ak", "gs.secret_key", "sk", + "fs.gs.auth.type", "UNAUTHENTICATED"))); + Assertions.assertDoesNotThrow(() -> GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.credential_provider_type", "ANONYMOUS", + "fs.gs.auth.type", "UNAUTHENTICATED"))); + GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( + "provider", "GCP", "gs.credential_provider_type", "COMPUTE_ENGINE", + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com", + "fs.gs.http.max.retry", "3")); + Assertions.assertEquals("COMPUTE_ENGINE", props.toHadoopConfigurationMap().get("fs.gs.auth.type")); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", + props.toHadoopConfigurationMap().get("fs.gs.auth.impersonation.service.account")); + } + @Test void explicitAnonymousUsesUnsignedBackendAndHadoop() { GcsFileSystemProperties props = GcsFileSystemProperties.of(Map.of( From b9f6a16375396a81880f8f1af35c2e2a5e3779ea Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 11:20:32 +0800 Subject: [PATCH 14/47] [fix](fe) Preserve native GCP paths and vault identities ### What problem does this PR solve? Related PR: #68540 Problem Summary: Hudi passed HMS s3/s3a locations to Hadoop while native GCP authentication only configured the gs filesystem. Normalize these locations when creating the Hudi table handle so metadata reads, scan planning, and JNI reads share the GCS base path. Preserve legacy HMAC and other filesystem paths. SHOW CREATE STORAGE VAULT omitted native GCP identity fields and emitted masked legacy keys that conflict with native authentication. Include the credential provider and optional impersonation account, and emit legacy keys only for legacy authentication. Add COW/MOR path coverage and DDL parse/credential round-trip tests with legacy masking checks. ### Release note Fix native GCP access to Hudi tables registered with s3/s3a locations and preserve native GCP identities in SHOW CREATE STORAGE VAULT. ### Check List (For Author) - Test: Unit tests added; FE Core/Hudi Checkstyle, connector import boundary, metadata funnel, and git diff --check passed. Standard FE test runner blocked by unusable local JDK 17. Hudi reactor tests blocked before test execution by missing required thirdparty Thrift compiler after dependency resolution. - Behavior changed: Yes, native GCP Hudi aliases select the gs filesystem and vault DDL includes its native identity without conflicting masked keys. - Does this need documentation: No, fixes the behavior already described by the feature. --- .../connector/hudi/HudiConnectorMetadata.java | 15 +++- .../connector/hudi/HudiTableTypeTest.java | 45 ++++++++++ .../ShowCreateStorageVaultCommand.java | 15 +++- .../ShowCreateStorageVaultCommandTest.java | 90 +++++++++++++++++++ 4 files changed, 162 insertions(+), 3 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 0599c6efb05eb6..7863b04b856e3f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -203,7 +203,7 @@ public Optional getTableHandle( return Optional.empty(); } HmsTableInfo tableInfo = hmsClient.getTable(dbName, tableName); - String location = tableInfo.getLocation(); + String location = hadoopTableLocation(tableInfo.getLocation()); String hudiTableType = detectHudiTableType(tableInfo); // Extract partition key names @@ -222,6 +222,19 @@ public Optional getTableHandle( .build()); } + private String hadoopTableLocation(String location) { + // Native GCP auth configures fs.gs.*, while HMS may retain an S3-compatible location. + // Normalize the handle so metadata, split planning and the JNI reader share the same base path. + if ("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem".equals(storageHadoopConfig.get("fs.gs.impl"))) { + if (location.regionMatches(true, 0, "s3://", 0, 5)) { + return "gs://" + location.substring(5); + } else if (location.regionMatches(true, 0, "s3a://", 0, 6)) { + return "gs://" + location.substring(6); + } + } + return location; + } + @Override public Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java index 718717b7c9f2e9..af33f7e10d7e65 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java @@ -85,6 +85,51 @@ public void testUnknownWhenNoHudiSignal() { detect("org.apache.hadoop.mapred.TextInputFormat", Collections.emptyMap())); } + @Test + public void testNativeGcpLocationsSelectGcsForCowAndMor() { + for (String authType : List.of("APPLICATION_DEFAULT", "COMPUTE_ENGINE")) { + for (String inputFormat : List.of("org.apache.hudi.hadoop.HoodieParquetInputFormat", + "org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat")) { + Map config = Map.of( + "fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + "fs.gs.auth.type", authType, + "fs.gs.auth.impersonation.service.account", "target@project.iam.gserviceaccount.com"); + for (String scheme : List.of("s3", "s3a", "S3", "S3A", "gs")) { + HudiTableHandle handle = tableHandle(scheme + "://bucket/table%20name/", inputFormat, config); + Assertions.assertEquals("gs://bucket/table%20name/", handle.getBasePath()); + } + for (String location : List.of("hdfs://namenode/table", "file:///tmp/table")) { + Assertions.assertEquals(location, tableHandle(location, inputFormat, config).getBasePath()); + } + } + } + } + + @Test + public void testLegacyLocationsKeepTheirScheme() { + // GCS HMAC and anonymous access use S3A; ordinary AWS/HDFS configs must also stay unchanged. + for (Map config : List.of(Collections.emptyMap(), + Map.of("fs.gs.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"), + Map.of("fs.s3a.access.key", "access-key", "fs.s3a.secret.key", "secret-key"))) { + for (String location : List.of("s3://bucket/table", "s3a://bucket/table", "gs://bucket/table")) { + Assertions.assertEquals(location, tableHandle(location, + "org.apache.hudi.hadoop.HoodieParquetInputFormat", config).getBasePath()); + } + } + } + + private HudiTableHandle tableHandle(String location, String inputFormat, Map config) { + HmsTableInfo info = HmsTableInfo.builder() + .dbName("db").tableName("t") + .location(location) + .inputFormat(inputFormat) + .parameters(Collections.emptyMap()) + .build(); + HudiConnectorMetadata metadata = new HudiConnectorMetadata(new FakeHmsClient(info), + HudiTestProperties.minimal(), new DirectHudiMetaClientExecutor(), config); + return (HudiTableHandle) metadata.getTableHandle(null, "db", "t").orElseThrow(); + } + /** * Minimal {@link HmsClient} double returning a fixed table. Only {@code tableExists} * and {@code getTable} are exercised by {@code getTableHandle}; the rest fail loud. diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java index 1c322f9d8b0e84..25cb81a5d62fbf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java @@ -28,6 +28,7 @@ import org.apache.doris.common.Config; import org.apache.doris.common.FeConstants; import org.apache.doris.common.util.DatasourcePrintableMap; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.mysql.privilege.AccessControllerManager; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.PlanType; @@ -130,8 +131,18 @@ private String getObjectCreateStmt(Cloud.ObjectStoreInfoPB objectInfo) { properties.put("s3.region", objectInfo.getRegion()); properties.put("s3.root.path", objectInfo.getPrefix()); properties.put("s3.bucket", objectInfo.getBucket()); - properties.put("s3.access_key", objectInfo.getAk()); - properties.put("s3.secret_key", objectInfo.getSk()); + if (objectInfo.hasCredential() && objectInfo.getCredential().hasGcpCredential()) { + Cloud.GcpCredentialPB credential = objectInfo.getCredential().getGcpCredential(); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); + if (!credential.getImpersonationServiceAccount().isEmpty()) { + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, + credential.getImpersonationServiceAccount()); + } + } else { + // Masking even empty keys would produce nonempty HMAC credentials conflicting with native auth. + properties.put("s3.access_key", objectInfo.getAk()); + properties.put("s3.secret_key", objectInfo.getSk()); + } properties.put("provider", objectInfo.getProvider().name()); properties.put("use_path_style", String.valueOf(objectInfo.getUsePathStyle())); if (objectInfo.hasExternalEndpoint()) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java new file mode 100644 index 00000000000000..450b3094b436c8 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java @@ -0,0 +1,90 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands; + +import org.apache.doris.cloud.proto.Cloud; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.common.util.DatasourcePrintableMap; +import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; +import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.nereids.parser.NereidsParser; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.List; +import java.util.Map; + +public class ShowCreateStorageVaultCommandTest { + @Test + public void testNativeGcpIdentityRoundTrip() { + for (Cloud.GcpCredentialPB.CredentialProviderType providerType : List.of( + Cloud.GcpCredentialPB.CredentialProviderType.DEFAULT, + Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE)) { + for (String account : List.of("", "target@project.iam.gserviceaccount.com")) { + Cloud.GcpCredentialPB.Builder credential = Cloud.GcpCredentialPB.newBuilder() + .setCredentialProviderType(providerType); + if (!account.isEmpty()) { + credential.setImpersonationServiceAccount(account); + } + Cloud.ObjectStoreInfoPB.Builder original = objectInfo(Cloud.ObjectStoreInfoPB.Provider.GCP); + original.getCredentialBuilder().setGcpCredential(credential); + + Map properties = shownProperties(original.build()); + Assertions.assertEquals(providerType.name(), properties.get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals(account, properties.getOrDefault(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "")); + Assertions.assertFalse(properties.containsKey("s3.access_key")); + Assertions.assertFalse(properties.containsKey("s3.secret_key")); + Assertions.assertEquals(original.build(), CloudObjectStoreAdapter.getObjStoreInfoPB(properties).build(), + "SHOW CREATE must preserve the native identity and connection properties"); + } + } + } + + @Test + public void testLegacyKeysRemainMasked() { + for (Cloud.ObjectStoreInfoPB.Provider provider : List.of( + Cloud.ObjectStoreInfoPB.Provider.GCP, Cloud.ObjectStoreInfoPB.Provider.S3)) { + Map properties = shownProperties(objectInfo(provider) + .setAk("access-key").setSk("secret-key").build()); + Assertions.assertEquals(DatasourcePrintableMap.PASSWORD_MASK, properties.get("s3.access_key")); + Assertions.assertEquals(DatasourcePrintableMap.PASSWORD_MASK, properties.get("s3.secret_key")); + Assertions.assertFalse(properties.containsKey(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertFalse(properties.containsKey(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + } + } + + private Cloud.ObjectStoreInfoPB.Builder objectInfo(Cloud.ObjectStoreInfoPB.Provider provider) { + return Cloud.ObjectStoreInfoPB.newBuilder() + .setProvider(provider) + .setEndpoint("storage.googleapis.com") + .setExternalEndpoint("storage.googleapis.com") + .setRegion("us-central1") + .setBucket("bucket") + .setPrefix("vault") + .setUsePathStyle(true); + } + + private Map shownProperties(Cloud.ObjectStoreInfoPB info) { + ShowCreateStorageVaultCommand command = new ShowCreateStorageVaultCommand("gcp_vault"); + String ddl = Deencapsulation.invoke(command, "getObjectCreateStmt", info); + CreateStorageVaultCommand parsed = (CreateStorageVaultCommand) new NereidsParser().parseSingle(ddl); + Assertions.assertEquals("gcp_vault", parsed.getVaultName()); + return parsed.getProperties(); + } +} From cacc8a4f023de66dd867896ba7d4c454e11b6dd2 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 12:38:30 +0800 Subject: [PATCH 15/47] [fix](storage) Complete native GCP scanner and credential paths ### What problem does this PR solve? Related PR: #68540 Problem Summary: Isolated BE JNI scanners cannot inherit GCS classes from the system Hadoop classpath. Bundle the shaded GCS connector with Hudi, Paimon and Iceberg scanners, and include it in the plugin dependency-closure inspection roots. Move the GCP Iceberg FileIO factory to fe-iceberg-common so the FE connector and BE metadata scanner ship the same factory and runtime dependencies. Preserve its fully qualified class name for serialized FileIO compatibility. DEFAULT ADC could read and upload objects but could not sign an error-log URL even when it selected a service account. Resolve the signer email from the same Google OAuth credentials that supply the token, retaining Google's ADC selection, caching and refresh implementation. Do not infer the identity from VM presence. Preserve explicit impersonation and reject credentials without a valid service account identity. An ALTER that only cleared impersonation on an HMAC vault implicitly selected ADC and erased the keys. Reject that patch without changing stored metadata; an explicit provider or nonempty target can still select native authentication. ### Release note Fix native GCP JNI scanner dependencies, Iceberg metadata-table FileIO reconstruction, DEFAULT ADC error-log signing and empty impersonation ALTER semantics. ### Check List (For Author) - Test: Added scanner filesystem loading and Iceberg FileIO serialization/client-reconstruction coverage, a signed URL flow with mocked IAM signing, account-selection cases, and HMAC-preservation assertions. Java Checkstyle, clang-format 16, Maven XML parsing, BE build hygiene, connector import boundaries, metadata funnel and git diff --check passed. FE/BE test scripts stopped at the local JDK 17 check; Cloud tests and live IAM tests were not run. clang-tidy could not run because no compilation database is available. - Behavior changed: Yes, as described above. - Does this need documentation: No, completes the documented native GCP behavior. Self-review: Changes are scoped to the four review findings. Token and account access share one credential object and mutex; ownership remains client-local without new initialization dependencies. No process config, SPI surface, persisted schema or FE/BE protocol changes are introduced. Existing credential modes and factory names are preserved. New empty-patch rejection precedes target mutation. Failure still returns an error/local error-log fallback; no additional metrics or hot-path allocation is needed. Rowset/version and transaction commit semantics are unchanged. Runtime and packaging acceptance still require CI and a configured GCP environment. --- be/src/io/fs/gcs_signed_url_provider.cpp | 59 +++------ be/src/io/fs/gcs_signed_url_provider.h | 5 + .../io/fs/gcs_signed_url_provider_test.cpp | 125 ++++++++++++++++++ cloud/src/common/auth/obj_credential.cpp | 5 + cloud/test/obj_credential_test.cpp | 6 + .../auth/gcp/gcp_token_provider.cpp | 19 ++- .../obj-client/auth/gcp/gcp_token_provider.h | 15 ++- .../hadoop-hudi-scanner/pom.xml | 7 + .../doris/hudi/HadoopHudiPluginTest.java | 11 ++ .../iceberg-metadata-scanner/pom.xml | 12 ++ .../doris/iceberg/IcebergPluginTest.java | 36 +++++ fe/be-java-extensions/paimon-scanner/pom.xml | 7 + .../apache/doris/paimon/PaimonPluginTest.java | 11 ++ fe/fe-connector/fe-connector-iceberg/pom.xml | 4 +- fe/fe-iceberg-common/pom.xml | 51 +++++++ .../iceberg/GcpS3FileIOAwsClientFactory.java | 0 fe/pom.xml | 1 + tools/be-java-plugins/check_plugin_layout.py | 2 + 18 files changed, 321 insertions(+), 55 deletions(-) create mode 100644 be/test/io/fs/gcs_signed_url_provider_test.cpp create mode 100644 fe/fe-iceberg-common/pom.xml rename fe/{fe-connector/fe-connector-iceberg => fe-iceberg-common}/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java (100%) diff --git a/be/src/io/fs/gcs_signed_url_provider.cpp b/be/src/io/fs/gcs_signed_url_provider.cpp index 22eddad6dcd6d9..79a040fa7523a6 100644 --- a/be/src/io/fs/gcs_signed_url_provider.cpp +++ b/be/src/io/fs/gcs_signed_url_provider.cpp @@ -20,16 +20,14 @@ #include #include -#include -#include #include #include #include #include "cpp/obj-client/auth/gcp/gcp_token_provider.h" #include "cpp/obj-client/auth/gcp/gcs_signed_url.h" +#include "cpp/sync_point.h" #include "service/http/http_client.h" -#include "util/string_util.h" #include "util/url_coding.h" namespace doris::io { @@ -37,10 +35,6 @@ namespace { constexpr std::string_view IAM_CREDENTIALS_ENDPOINT = "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/"; -constexpr std::string_view DEFAULT_METADATA_HOST = "metadata.google.internal"; -constexpr std::string_view METADATA_SERVICE_ACCOUNT_EMAIL_PATH = - "/computeMetadata/v1/instance/service-accounts/default/email"; -constexpr int64_t METADATA_REQUEST_TIMEOUT_MS = 1000; std::string iam_error_message(const rapidjson::Document& document) { if (!document.IsObject() || !document.HasMember("error") || !document["error"].IsObject()) { @@ -92,6 +86,8 @@ std::string percent_encode(std::string_view value) { Status call_iam_sign_blob(std::string_view access_token, std::string_view service_account, std::string_view string_to_sign, int64_t request_timeout_ms, std::string* signature) { + TEST_SYNC_POINT_RETURN_WITH_VALUE("GcsV4Signer::sign_blob", Status::OK(), access_token, + service_account, string_to_sign, signature); std::string encoded_payload; base64_encode(std::string(string_to_sign), &encoded_payload); std::string request_body = fmt::format(R"({{"payload":"{}"}})", encoded_payload); @@ -126,33 +122,21 @@ Status call_iam_sign_blob(std::string_view access_token, std::string_view servic return Status::OK(); } -Status fetch_metadata_service_account_email(int64_t request_timeout_ms, std::string* email) { - const char* configured_host = std::getenv("GCE_METADATA_HOST"); - std::string_view metadata_host = configured_host != nullptr && configured_host[0] != '\0' - ? configured_host - : DEFAULT_METADATA_HOST; - HttpClient client; - RETURN_IF_ERROR(client.init( - fmt::format("http://{}{}", metadata_host, METADATA_SERVICE_ACCOUNT_EMAIL_PATH), true, - HttpClient::AuthTokenMode::NONE)); - client.set_header("Metadata-Flavor", "Google"); - client.set_timeout_ms(request_timeout_ms > 0 - ? std::min(request_timeout_ms, METADATA_REQUEST_TIMEOUT_MS) - : METADATA_REQUEST_TIMEOUT_MS); +} // namespace - std::string response; - RETURN_IF_ERROR(client.execute(&response)); - auto resolved_email = trim(response); - if (!is_valid_gcp_service_account_email(resolved_email)) { - return Status::InternalError( - "GCP metadata server returned an invalid service account email"); +Status resolve_gcs_signer_email(const GcpCredentialConfig& credential, + const GcpTokenProvider& token_provider, std::string* signer_email) { + *signer_email = credential.impersonation_service_account.empty() + ? token_provider.get_service_account_email() + : credential.impersonation_service_account; + if (!is_valid_gcp_service_account_email(*signer_email)) { + return Status::InvalidArgument( + "GCS V4 signing requires a service account identity; configure " + "gs.impersonation_service_account for credentials without one"); } - email->assign(resolved_email); return Status::OK(); } -} // namespace - Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, const GcpCredentialConfig& credential, const std::shared_ptr& token_provider, @@ -162,18 +146,6 @@ Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, } signed_url->clear(); - std::string signer_email; - if (!credential.impersonation_service_account.empty()) { - signer_email = credential.impersonation_service_account; - } else if (credential.provider_type == GcpCredentialProviderType::ComputeEngine) { - RETURN_IF_ERROR( - fetch_metadata_service_account_email(options.request_timeout_ms, &signer_email)); - } else { - return Status::InvalidArgument( - "GCS V4 signing with DEFAULT credentials requires " - "gs.impersonation_service_account; use COMPUTE_ENGINE to resolve the VM service " - "account from metadata"); - } if (token_provider == nullptr) { return Status::InternalError("GCS V4 signing token provider is not initialized"); } @@ -184,6 +156,11 @@ Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, return Status::InternalError("failed to obtain OAuth token for IAM signBlob"); } + // Resolve identity from the same credentials that supplied the token. DEFAULT may + // select a VM, a key file, or another ADC source; never infer it from VM presence. + std::string signer_email; + RETURN_IF_ERROR(resolve_gcs_signer_email(credential, *token_provider, &signer_email)); + auto result = build_gcs_v4_signed_url( {.endpoint = options.endpoint, .bucket = options.bucket, diff --git a/be/src/io/fs/gcs_signed_url_provider.h b/be/src/io/fs/gcs_signed_url_provider.h index 771a247bf2d112..c925e758042d09 100644 --- a/be/src/io/fs/gcs_signed_url_provider.h +++ b/be/src/io/fs/gcs_signed_url_provider.h @@ -38,6 +38,11 @@ struct GcsV4SignedUrlProviderOptions { int64_t request_timeout_ms = 10000; }; +// Resolve the identity selected by ADC or explicit impersonation, without falling back +// to an unrelated VM service account when the selected credentials have no identity. +Status resolve_gcs_signer_email(const GcpCredentialConfig& credential, + const GcpTokenProvider& token_provider, std::string* signer_email); + // Resolves the signing service account, obtains an OAuth token, and invokes // IAM Credentials signBlob before delegating URL construction to common auth. Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, diff --git a/be/test/io/fs/gcs_signed_url_provider_test.cpp b/be/test/io/fs/gcs_signed_url_provider_test.cpp new file mode 100644 index 00000000000000..98b6f7b134a593 --- /dev/null +++ b/be/test/io/fs/gcs_signed_url_provider_test.cpp @@ -0,0 +1,125 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "io/fs/gcs_signed_url_provider.h" + +#include + +#include +#include + +#include "cpp/obj-client/auth/gcp/gcp_token_provider.h" +#include "cpp/sync_point.h" + +namespace doris::io { +namespace { + +class SelectedAccountCredentials : public google::cloud::oauth2_internal::Credentials { +public: + explicit SelectedAccountCredentials(std::string email) : _email(std::move(email)) {} + + google::cloud::StatusOr GetToken( + std::chrono::system_clock::time_point now) override { + return google::cloud::AccessToken {"test-token", now + std::chrono::hours(1)}; + } + + std::string AccountEmail() const override { return _email; } + +private: + std::string _email; +}; + +class GcsSignedUrlProviderTest : public testing::Test { +protected: + void TearDown() override { + SyncPoint::get_instance()->clear_call_back("GcsV4Signer::sign_blob"); + SyncPoint::get_instance()->disable_processing(); + } +}; + +TEST_F(GcsSignedUrlProviderTest, DefaultSignsErrorLogWithResolvedIdentity) { + auto tokens = std::make_shared( + std::make_shared("vm@my-project.iam.gserviceaccount.com")); + bool called = false; + SyncPoint::get_instance()->set_call_back( + "GcsV4Signer::sign_blob", [&](std::vector&& args) { + called = true; + EXPECT_EQ(try_any_cast(args[0]), "test-token"); + EXPECT_EQ(try_any_cast(args[1]), + "vm@my-project.iam.gserviceaccount.com"); + EXPECT_EQ(try_any_cast(args[2]).find("GOOG4-RSA-SHA256"), 0); + *try_any_cast(args[3]) = "test-signature"; + auto* result = try_any_cast_ret(args); + result->second = true; + }); + SyncPoint::get_instance()->enable_processing(); + GcpCredentialConfig credential; + credential.provider_type = GcpCredentialProviderType::Default; + GcsV4SignedUrlProviderOptions options {.endpoint = "storage.googleapis.com", + .bucket = "bucket", + .key = "load-errors/error.log", + .expiration_secs = 300}; + std::string url; + auto status = generate_gcs_v4_signed_url(options, credential, tokens, &url); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(called); + EXPECT_NE(url.find("vm%40my-project.iam.gserviceaccount.com"), std::string::npos); + EXPECT_NE(url.find("X-Goog-Signature="), std::string::npos); +} + +TEST_F(GcsSignedUrlProviderTest, DefaultAndComputeUseTheSelectedAccount) { + // Represents ADC selecting either the VM account or a service-account file. + // Both must use the account attached to the token, without a separate metadata lookup. + for (auto source : + {GcpCredentialProviderType::Default, GcpCredentialProviderType::ComputeEngine}) { + for (const auto* email : + {"vm@my-project.iam.gserviceaccount.com", "file@my-project.iam.gserviceaccount.com"}) { + GcpTokenProvider tokens(std::make_shared(email)); + EXPECT_EQ(tokens.get_token(), "test-token"); + GcpCredentialConfig credential; + credential.provider_type = source; + std::string signer; + ASSERT_TRUE(resolve_gcs_signer_email(credential, tokens, &signer).ok()); + EXPECT_EQ(signer, email); + } + } +} + +TEST_F(GcsSignedUrlProviderTest, AccountlessAdcRequiresExplicitImpersonation) { + GcpTokenProvider tokens(std::make_shared("")); + GcpCredentialConfig credential; + credential.provider_type = GcpCredentialProviderType::Default; + std::string signer; + EXPECT_FALSE(resolve_gcs_signer_email(credential, tokens, &signer).ok()); + + credential.impersonation_service_account = "target@my-project.iam.gserviceaccount.com"; + ASSERT_TRUE(resolve_gcs_signer_email(credential, tokens, &signer).ok()); + EXPECT_EQ(signer, credential.impersonation_service_account); +} + +TEST_F(GcsSignedUrlProviderTest, ImpersonationOverridesSourceAccount) { + GcpTokenProvider tokens(std::make_shared( + "source@my-project.iam.gserviceaccount.com")); + GcpCredentialConfig credential; + credential.impersonation_service_account = "target@my-project.iam.gserviceaccount.com"; + std::string signer; + ASSERT_TRUE(resolve_gcs_signer_email(credential, tokens, &signer).ok()); + EXPECT_EQ(signer, credential.impersonation_service_account); +} + +} // namespace +} // namespace doris::io diff --git a/cloud/src/common/auth/obj_credential.cpp b/cloud/src/common/auth/obj_credential.cpp index acb7116c1bc25c..e2db4815bc56ed 100644 --- a/cloud/src/common/auth/obj_credential.cpp +++ b/cloud/src/common/auth/obj_credential.cpp @@ -202,6 +202,11 @@ std::optional apply_obj_credential(const ObjectStoreInfoPB& update, if (!patch.has_credential_provider_type() && !patch.has_impersonation_service_account()) { return "GCP credential update must specify a provider type or service account"; } + if (!target->credential().has_gcp_credential() && !patch.has_credential_provider_type() && + patch.impersonation_service_account().empty()) { + return "clearing impersonation requires an existing native GCP credential; " + "specify a provider type to replace HMAC credentials"; + } auto* credential = candidate.mutable_credential()->mutable_gcp_credential(); if (target->credential().has_gcp_credential()) { credential->CopyFrom(target->credential().gcp_credential()); diff --git a/cloud/test/obj_credential_test.cpp b/cloud/test/obj_credential_test.cpp index 50905efec6dd48..9f33e5ab219139 100644 --- a/cloud/test/obj_credential_test.cpp +++ b/cloud/test/obj_credential_test.cpp @@ -82,6 +82,7 @@ TEST(ObjCredentialTest, SwitchFromStaticDefaultsSourceAndRemovesLegacyCredential stored.set_ak("encrypted-ak"); stored.set_sk("encrypted-sk"); stored.mutable_encryption_info()->set_key_id(1); + const auto original = stored.SerializeAsString(); ObjectStoreInfoPB update; auto* credential = update.mutable_credential()->mutable_gcp_credential(); credential->set_impersonation_service_account(account); @@ -89,6 +90,11 @@ TEST(ObjCredentialTest, SwitchFromStaticDefaultsSourceAndRemovesLegacyCredential credential->set_credential_provider_type(GcpCredentialPB::COMPUTE_ENGINE); } auto error = apply_obj_credential(update, &stored); + if (!explicit_source && std::string(account).empty()) { + ASSERT_TRUE(error.has_value()); + EXPECT_EQ(stored.SerializeAsString(), original); + continue; + } ASSERT_FALSE(error.has_value()) << error.value_or(""); EXPECT_TRUE(stored.credential().gcp_credential().has_credential_provider_type()); EXPECT_EQ(stored.credential().gcp_credential().credential_provider_type(), diff --git a/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp b/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp index 596c4370dac4f9..9716f786a3a384 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp @@ -19,7 +19,7 @@ #include #include -#include +#include #include @@ -71,8 +71,12 @@ std::shared_ptr make_credentials(const GcpCredential GcpTokenProvider::GcpTokenProvider(const GcpCredentialConfig& credential, const std::string& ca_cert_path, GcpTokenScope token_scope) - : _credentials(make_credentials(credential, ca_cert_path, token_scope)), - _token_generator(google::cloud::oauth2::MakeAccessTokenGenerator(*_credentials)) {} + : GcpTokenProvider(google::cloud::rest_internal::MapCredentials( + *make_credentials(credential, ca_cert_path, token_scope))) {} + +GcpTokenProvider::GcpTokenProvider( + std::shared_ptr credentials) + : _credentials(std::move(credentials)) {} GcpTokenProvider::~GcpTokenProvider() = default; @@ -80,8 +84,8 @@ std::optional GcpTokenProvider::get_token() const { // GcpS3Client and presign callers may share one provider across request // threads. Serialize access even if the underlying implementation changes // its thread-safety guarantees in a future google-cloud-cpp release. - std::lock_guard lock(_token_generator_mutex); - auto token = _token_generator->GetToken(); + std::lock_guard lock(_credentials_mutex); + auto token = _credentials->GetToken(std::chrono::system_clock::now()); if (!token) { LOG_EVERY_N(WARNING, 100) << "failed to obtain GCP access token: " << token.status().message(); @@ -90,4 +94,9 @@ std::optional GcpTokenProvider::get_token() const { return token->token; } +std::string GcpTokenProvider::get_service_account_email() const { + std::lock_guard lock(_credentials_mutex); + return _credentials->AccountEmail(); +} + } // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_token_provider.h b/common/cpp/obj-client/auth/gcp/gcp_token_provider.h index 8c39aa2b2dcb90..eb94166cc3fec4 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_token_provider.h +++ b/common/cpp/obj-client/auth/gcp/gcp_token_provider.h @@ -17,8 +17,7 @@ #pragma once -#include -#include +#include #include #include @@ -34,24 +33,26 @@ enum class GcpTokenScope { CloudPlatform, }; -// Owns both objects because MakeAccessTokenGenerator takes Credentials by const -// reference. The Google auth library handles token caching and refresh internally. +// Use one resolved credential for tokens and signing identity. The Google auth +// library handles ADC selection, token caching and refresh internally. class GcpTokenProvider { public: explicit GcpTokenProvider(const GcpCredentialConfig& credential, const std::string& ca_cert_path = "", GcpTokenScope token_scope = GcpTokenScope::StorageReadWrite); + explicit GcpTokenProvider( + std::shared_ptr credentials); ~GcpTokenProvider(); GcpTokenProvider(const GcpTokenProvider&) = delete; GcpTokenProvider& operator=(const GcpTokenProvider&) = delete; std::optional get_token() const; + std::string get_service_account_email() const; private: - std::shared_ptr _credentials; - std::shared_ptr _token_generator; - mutable std::mutex _token_generator_mutex; + std::shared_ptr _credentials; + mutable std::mutex _credentials_mutex; }; } // namespace doris diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml b/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml index 525d763030bc23..835b2dff5afefa 100644 --- a/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml +++ b/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml @@ -43,6 +43,13 @@ under the License. + + + com.google.cloud.bigdataoss + gcs-connector + shaded + runtime + org.apache.doris jni-spi diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java b/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java index bc40bef8eee19f..a099fc46626c53 100644 --- a/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java +++ b/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java @@ -111,6 +111,16 @@ private static DorisPlugin loadPlugin() { } /** The path the plugin registry takes: services file, plugin class, factory list. */ + @Test + public void instantiatesNativeGcsFilesystem() throws Exception { + Configuration conf = new Configuration(); + conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + try (FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance()) { + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + fs.getClass().getName()); + } + } + @Test public void isDiscoverableThroughServiceLoader() { Assertions.assertTrue(loadPlugin() instanceof HadoopHudiPlugin); @@ -164,6 +174,7 @@ public void resolvesEveryFilesystemSchemeAScanCanArriveOn() throws IOException { expected.put("hdfs", "org.apache.hadoop.hdfs.DistributedFileSystem"); expected.put("webhdfs", "org.apache.hadoop.hdfs.web.WebHdfsFileSystem"); expected.put("s3a", "org.apache.hadoop.fs.s3a.S3AFileSystem"); + expected.put("gs", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); expected.put("obs", "org.apache.hadoop.fs.obs.OBSFileSystem"); expected.put("abfs", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem"); expected.put("wasb", "org.apache.hadoop.fs.azure.NativeAzureFileSystem"); diff --git a/fe/be-java-extensions/iceberg-metadata-scanner/pom.xml b/fe/be-java-extensions/iceberg-metadata-scanner/pom.xml index b6d275327ae4ad..61eca21ff267dd 100644 --- a/fe/be-java-extensions/iceberg-metadata-scanner/pom.xml +++ b/fe/be-java-extensions/iceberg-metadata-scanner/pom.xml @@ -41,6 +41,18 @@ under the License. + + + com.google.cloud.bigdataoss + gcs-connector + shaded + runtime + + + org.apache.doris + fe-iceberg-common + ${revision} + org.apache.doris jni-spi diff --git a/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java b/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java index 0b82ae6f898779..cf7ce236cc753b 100644 --- a/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java +++ b/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java @@ -32,6 +32,8 @@ import org.apache.iceberg.Schema; import org.apache.iceberg.Table; import org.apache.iceberg.TableScan; +import org.apache.iceberg.aws.s3.S3FileIO; +import org.apache.iceberg.aws.s3.S3FileIOProperties; import org.apache.iceberg.hadoop.HadoopTables; import org.apache.iceberg.io.CloseableIterable; import org.apache.iceberg.types.Types; @@ -88,6 +90,39 @@ private static DorisPlugin loadPlugin() { } /** The path the plugin registry takes: services file, plugin class, factory list. */ + @Test + public void reconstructsNativeGcpFileIoAfterSerialization() throws Exception { + Map properties = new HashMap<>(); + properties.put(S3FileIOProperties.CLIENT_FACTORY, + "org.apache.doris.connector.iceberg.GcpS3FileIOAwsClientFactory"); + properties.put("provider", "GCP"); + properties.put("s3.endpoint", "https://storage.googleapis.com"); + properties.put("s3.region", "us-central1"); + properties.put("gs.credential_provider_type", "COMPUTE_ENGINE"); + properties.put("gs.impersonation_service_account", "target@project.iam.gserviceaccount.com"); + try (S3FileIO original = new S3FileIO()) { + original.initialize(properties); + // Metadata tasks carry this FileIO across to BE, where the first manifest read rebuilds its client. + String wire = SerializationUtil.serializeToBase64(original); + try (S3FileIO restored = IcebergSerializationCompat.deserializeFromBase64(wire)) { + Assertions.assertEquals(properties, restored.properties()); + Assertions.assertNotNull(restored.client()); + Assertions.assertEquals("s3://bucket/manifest.avro", + restored.newInputFile("s3://bucket/manifest.avro").location()); + } + } + } + + @Test + public void instantiatesNativeGcsFilesystem() throws Exception { + Configuration conf = new Configuration(); + conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + try (FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance()) { + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + fs.getClass().getName()); + } + } + @Test public void isDiscoverableThroughServiceLoader() { Assertions.assertTrue(loadPlugin() instanceof IcebergPlugin); @@ -132,6 +167,7 @@ public void resolvesEveryFilesystemSchemeAScanCanArriveOn() throws IOException { expected.put("hdfs", "org.apache.hadoop.hdfs.DistributedFileSystem"); expected.put("webhdfs", "org.apache.hadoop.hdfs.web.WebHdfsFileSystem"); expected.put("s3a", "org.apache.hadoop.fs.s3a.S3AFileSystem"); + expected.put("gs", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); expected.put("obs", "org.apache.hadoop.fs.obs.OBSFileSystem"); expected.put("abfs", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem"); expected.put("wasb", "org.apache.hadoop.fs.azure.NativeAzureFileSystem"); diff --git a/fe/be-java-extensions/paimon-scanner/pom.xml b/fe/be-java-extensions/paimon-scanner/pom.xml index 1f771a704ed5ba..2dbb8cbc028deb 100644 --- a/fe/be-java-extensions/paimon-scanner/pom.xml +++ b/fe/be-java-extensions/paimon-scanner/pom.xml @@ -41,6 +41,13 @@ under the License. + + + com.google.cloud.bigdataoss + gcs-connector + shaded + runtime + org.apache.doris jni-spi diff --git a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java index 0206714c03e5db..c9ea6273b2e1d8 100644 --- a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java +++ b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java @@ -93,6 +93,16 @@ private static DorisPlugin loadPlugin() { } /** The path the plugin registry takes: services file, plugin class, factory list. */ + @Test + public void instantiatesNativeGcsFilesystem() throws Exception { + Configuration conf = new Configuration(); + conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + try (FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance()) { + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + fs.getClass().getName()); + } + } + @Test public void isDiscoverableThroughServiceLoader() { Assertions.assertTrue(loadPlugin() instanceof PaimonPlugin); @@ -145,6 +155,7 @@ public void resolvesEveryFilesystemSchemeAScanCanArriveOn() throws IOException { expected.put("hdfs", "org.apache.hadoop.hdfs.DistributedFileSystem"); expected.put("webhdfs", "org.apache.hadoop.hdfs.web.WebHdfsFileSystem"); expected.put("s3a", "org.apache.hadoop.fs.s3a.S3AFileSystem"); + expected.put("gs", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); expected.put("obs", "org.apache.hadoop.fs.obs.OBSFileSystem"); expected.put("abfs", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem"); expected.put("wasb", "org.apache.hadoop.fs.azure.NativeAzureFileSystem"); diff --git a/fe/fe-connector/fe-connector-iceberg/pom.xml b/fe/fe-connector/fe-connector-iceberg/pom.xml index 922dc0a7fa3afa..e474b0cf8d7f63 100644 --- a/fe/fe-connector/fe-connector-iceberg/pom.xml +++ b/fe/fe-connector/fe-connector-iceberg/pom.xml @@ -41,10 +41,10 @@ under the License. - + org.apache.doris - fe-filesystem-gcs + fe-iceberg-common ${revision} diff --git a/fe/fe-iceberg-common/pom.xml b/fe/fe-iceberg-common/pom.xml new file mode 100644 index 00000000000000..d139e7aa15bef2 --- /dev/null +++ b/fe/fe-iceberg-common/pom.xml @@ -0,0 +1,51 @@ + + + + 4.0.0 + + org.apache.doris + fe + ${revision} + ../pom.xml + + fe-iceberg-common + Doris Shared Iceberg FileIO + Native GCP FileIO factory shared by the FE Iceberg connector and the isolated + BE metadata scanner. Its class name is part of serialized Iceberg scan tasks. + + + org.apache.iceberg + iceberg-aws + ${iceberg.version} + + + org.apache.doris + fe-filesystem-gcs + ${revision} + + + + org.apache.doris + fe-filesystem-spi + ${revision} + + + diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java b/fe/fe-iceberg-common/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java similarity index 100% rename from fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java rename to fe/fe-iceberg-common/src/main/java/org/apache/doris/connector/iceberg/GcpS3FileIOAwsClientFactory.java diff --git a/fe/pom.xml b/fe/pom.xml index 687496d0c582b5..41b2c314a4d915 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -230,6 +230,7 @@ under the License. fe-kerberos fe-common fe-trino-connector-common + fe-iceberg-common fe-catalog fe-filesystem fe-extension-spi diff --git a/tools/be-java-plugins/check_plugin_layout.py b/tools/be-java-plugins/check_plugin_layout.py index e8576b9246d3ee..e63510e01f8a91 100755 --- a/tools/be-java-plugins/check_plugin_layout.py +++ b/tools/be-java-plugins/check_plugin_layout.py @@ -456,6 +456,8 @@ def _doris_owned(jars): # Entries are class file paths. A name absent from a plugin costs nothing: that plugin simply # bundles no such filesystem, and the root set is what it was. _NAMED_BY_DORIS = ( + # Native GCP Hadoop access in the isolated Hudi, Paimon and Iceberg scanners. + "com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystem.class", # fs.s3.impl / fs.s3a.impl / fs.cos.impl / fs.cosn.impl / fs.gs.impl, and the fallback # fs.obs.impl - one jar, hadoop-aws, and the S3A credential providers live in it too. "org/apache/hadoop/fs/s3a/S3AFileSystem.class", From 2845cc628ecea776f77218bc80ed890a1abf909c Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 14:47:15 +0800 Subject: [PATCH 16/47] [fix](fe) Align GCS authentication and Paimon URI handling ### What problem does this PR solve? Related PR: #68540 Problem Summary: Preserve the default credential chain for explicitly selected S3 storage at GCS endpoints instead of publishing anonymous authentication after an FE validity check. Infer GCP before normalizing resource and vault gs.* aliases, including partial resource updates whose stored provider was omitted. Normalize native-GCS Paimon warehouse and persisted file URI aliases at the FileIO boundary. Share the serializable FileIO implementation between the FE connector and isolated BE scanner so planning and JNI reads use fs.gs auth. Reject raw fs.gs.token.server.url overrides of the native credential flow. ### Release note Fix default S3 authentication at GCS endpoints, inferred GCP resource/vault creation, native GCS Paimon URI aliases, and token endpoint override validation. ### Check List (For Author) - Test: Unit tests added for all four cases. Passed: GcsAuthResolverTest (6 tests), NativeGcsFileIOTest (2 tests), Checkstyle for all six affected Java modules, FE connector import and metadata-funnel gates, git diff --check. NativeGcsFileIO tests exercise read/write/list/rename/delete for gs, s3, and s3a before and after Java serialization using local backing storage. Not completed: full FE build/UT require a working JDK 17; Paimon connector package tests stop at missing thirdparty Thrift 0.24; GCS property tests stop at a Maven TLS failure downloading netty-codec-http 4.2.17.Final. Live GCP IAM and packaged BE plugin execution were not available. - Behavior changed: Yes, as described above. - Does this need documentation: No additional properties are introduced. Self-review checkpoints: - Goal/proof: All four reported branches corrected; targeted executed and unexecuted tests distinguished above. - Scope/reuse: Reuse HadoopFileIO and existing auth/normalization boundaries; shared Paimon module is required by FE-to-BE FileIO serialization. - Concurrency: No new shared mutable state or lock ordering; existing resource ALTER synchronization and HadoopFileIO filesystem lifecycle are retained. - Lifecycle/static initialization: No new global initialization dependencies; serialized FileIO retains Hadoop configuration and lazy filesystem creation. - Configuration/dynamics: No new public properties; resource updates normalize before validation/publication. Native config restrictions apply at binding. - Compatibility: Explicit S3 default chain and HMAC paths preserved. The new native FileIO requires its shared jar in both updated FE and BE plugins. - Parallel paths: Filesystem/JDBC/HMS contexts and BE deserialization covered; REST continues to use server-managed storage. - Conditions: GCS URI rewriting requires native fs.gs implementation; other URI schemes and legacy S3A/HMAC bindings retain their existing behavior. - Test coverage: Includes positive, negative, alias, and serialization cases; environment limitations listed above. No golden output files edited. - Observability: Existing failures remain explicit; rejected token key is named. - Persistence: Canonical resource properties are published through existing persistence paths; no new metadata field or edit-log format. - Writes/atomicity: No transaction or rowset visibility changes; FileIO operations delegate after URI normalization. - FE/BE transport: Both plugins depend on the same shared FileIO class. - Performance: Constant-time scheme checks; retain HadoopFileIO caching. - Other concerns: No additional defect substantiated within these fixes. --- fe/be-java-extensions/paimon-scanner/pom.xml | 6 + .../apache/doris/paimon/PaimonPluginTest.java | 14 +++ fe/fe-connector/fe-connector-paimon/pom.xml | 6 + .../paimon/PaimonCatalogFactory.java | 14 +++ .../connector/paimon/PaimonConnector.java | 8 +- .../paimon/PaimonCatalogFactoryTest.java | 25 ++++ .../org/apache/doris/catalog/S3Resource.java | 14 ++- .../apache/doris/catalog/S3ResourceTest.java | 79 +++++++++++- .../filesystem/auth/GcsAuthResolver.java | 9 +- .../filesystem/auth/GcsAuthResolverTest.java | 14 +-- .../gcs/GcsFileSystemProperties.java | 5 +- .../gcs/GcsFileSystemPropertiesTest.java | 3 +- fe/fe-paimon-common/pom.xml | 50 ++++++++ .../apache/doris/paimon/NativeGcsFileIO.java | 117 ++++++++++++++++++ .../doris/paimon/NativeGcsFileIOTest.java | 106 ++++++++++++++++ fe/pom.xml | 1 + 16 files changed, 447 insertions(+), 24 deletions(-) create mode 100644 fe/fe-paimon-common/pom.xml create mode 100644 fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java create mode 100644 fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java diff --git a/fe/be-java-extensions/paimon-scanner/pom.xml b/fe/be-java-extensions/paimon-scanner/pom.xml index 2dbb8cbc028deb..094fd362ff80c0 100644 --- a/fe/be-java-extensions/paimon-scanner/pom.xml +++ b/fe/be-java-extensions/paimon-scanner/pom.xml @@ -41,6 +41,12 @@ under the License. + + + org.apache.doris + fe-paimon-common + ${revision} + com.google.cloud.bigdataoss diff --git a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java index c9ea6273b2e1d8..531ee496c3d2c6 100644 --- a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java +++ b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java @@ -93,6 +93,20 @@ private static DorisPlugin loadPlugin() { } /** The path the plugin registry takes: services file, plugin class, factory list. */ + @Test + public void reconstructsNativeGcsFileIO() throws Exception { + NativeGcsFileIO fileIO = new NativeGcsFileIO(); + Configuration conf = new Configuration(false); + conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + conf.set("fs.gs.auth.type", "APPLICATION_DEFAULT"); + fileIO.configure(CatalogContext.create(new Options(), conf)); + NativeGcsFileIO restored = InstantiationUtil.deserializeObject( + InstantiationUtil.serializeObject(fileIO), getClass().getClassLoader()); + Assertions.assertEquals("APPLICATION_DEFAULT", restored.hadoopConf().get("fs.gs.auth.type")); + Assertions.assertEquals("gs://bucket/file", NativeGcsFileIO.normalize( + new org.apache.paimon.fs.Path("s3a://bucket/file")).toString()); + } + @Test public void instantiatesNativeGcsFilesystem() throws Exception { Configuration conf = new Configuration(); diff --git a/fe/fe-connector/fe-connector-paimon/pom.xml b/fe/fe-connector/fe-connector-paimon/pom.xml index 9c332e6c19b465..1686a3c38de3ec 100644 --- a/fe/fe-connector/fe-connector-paimon/pom.xml +++ b/fe/fe-connector/fe-connector-paimon/pom.xml @@ -41,6 +41,12 @@ under the License. + + + org.apache.doris + fe-paimon-common + ${revision} + ${project.groupId} fe-connector-spi diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java index e6c96cdab9d4ec..a8fe652967669b 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java @@ -23,11 +23,13 @@ import org.apache.doris.connector.metastore.paimon.rest.PaimonRestMetaStoreProperties; import org.apache.doris.connector.metastore.spi.AbstractMetaStoreProperties; import org.apache.doris.connector.metastore.spi.MetaStoreProviders; +import org.apache.doris.paimon.NativeGcsFileIO; import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.paimon.catalog.CatalogContext; import org.apache.paimon.catalog.FileSystemCatalogFactory; import org.apache.paimon.jdbc.JdbcCatalogFactory; import org.apache.paimon.options.CatalogOptions; @@ -111,6 +113,18 @@ public static Options buildCatalogOptions(PaimonCatalogProperties catalogPropert return options; } + static CatalogContext createCatalogContext(Options options, Configuration conf) { + if (!"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem".equals(conf.get("fs.gs.impl"))) { + return CatalogContext.create(options, conf); + } + String warehouse = options.get(CatalogOptions.WAREHOUSE); + if (warehouse != null) { + options.set(CatalogOptions.WAREHOUSE, + NativeGcsFileIO.normalize(new org.apache.paimon.fs.Path(warehouse)).toString()); + } + return CatalogContext.create(options, conf, new NativeGcsFileIO.Loader(), null); + } + static boolean isCatalogCacheEnabled(PaimonCatalogProperties catalogProperties) { return assembleCatalogOptions(catalogProperties).get(CatalogOptions.CACHE_ENABLED); } diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java index 845ec5a025f2c1..d79500a0da3763 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java @@ -430,7 +430,7 @@ private Catalog createCatalog() { // filesystem carries a Hadoop Configuration for HDFS/S3 storage. Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration( catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(CatalogContext.create(options, conf), flavor, + return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, conf), flavor, "Failed to create Paimon catalog with filesystem metastore"); } case PaimonCatalogProperties.REST: { @@ -442,7 +442,7 @@ private Catalog createCatalog() { maybeRegisterJdbcDriver(); Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration( catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(CatalogContext.create(options, conf), flavor, + return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, conf), flavor, "Failed to create Paimon catalog with JDBC metastore"); } case PaimonCatalogProperties.HMS: { @@ -472,7 +472,7 @@ private Catalog createCatalog() { options.set("client-pool-cache.keys", appendHmsCacheKeys( options.get("client-pool-cache.keys"))); HadoopAuthenticator hmsAuth = buildHmsAuthenticator(catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(CatalogContext.create(options, hc), flavor, + return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, hc), flavor, hmsAuth, storageHadoopConfig, "Failed to create Paimon catalog with HMS metastore"); } @@ -489,7 +489,7 @@ private Catalog createCatalog() { dlfConf.put(PaimonCatalogFactory.DLF_CLIENT_POOL_IDENTITY, PaimonCatalogFactory.dlfClientPoolIdentity(dlfConf)); HiveConf hc = PaimonCatalogFactory.assembleHiveConf(null, dlfConf); - return createCatalogFromContext(CatalogContext.create(options, hc), flavor, + return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, hc), flavor, "Failed to create Paimon catalog with DLF metastore"); } default: diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java index 525f4855b9fa0f..87a9e1ab788cda 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java @@ -21,9 +21,11 @@ import org.apache.doris.filesystem.properties.HadoopStorageProperties; import org.apache.doris.filesystem.properties.StorageKind; import org.apache.doris.filesystem.properties.StorageProperties; +import org.apache.doris.paimon.NativeGcsFileIO; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.paimon.catalog.CatalogContext; import org.apache.paimon.options.CatalogOptions; import org.apache.paimon.options.Options; import org.junit.jupiter.api.Assertions; @@ -80,6 +82,29 @@ private static Map storage(String... kv) { // buildCatalogOptions — per-flavor metastore identifier + warehouse // --------------------------------------------------------------------- + @Test + public void nativeGcsNormalizesWarehouseAndInstallsAliasFileIO() { + for (String scheme : new String[] {"s3", "s3a", "gs"}) { + Options options = new Options(); + options.set(CatalogOptions.WAREHOUSE, scheme + "://bucket/private-warehouse"); + Configuration conf = new Configuration(false); + conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); + conf.set("fs.gs.auth.type", "APPLICATION_DEFAULT"); + CatalogContext context = PaimonCatalogFactory.createCatalogContext(options, conf); + Assertions.assertEquals("gs://bucket/private-warehouse", context.options().get(CatalogOptions.WAREHOUSE)); + Assertions.assertInstanceOf(NativeGcsFileIO.class, + context.preferIO().load(new org.apache.paimon.fs.Path(scheme + "://bucket/persisted-file"))); + Assertions.assertEquals("APPLICATION_DEFAULT", context.hadoopConf().get("fs.gs.auth.type")); + } + Options legacy = new Options(); + legacy.set(CatalogOptions.WAREHOUSE, "s3a://bucket/warehouse"); + Configuration conf = new Configuration(false); + conf.set("fs.gs.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"); + CatalogContext context = PaimonCatalogFactory.createCatalogContext(legacy, conf); + Assertions.assertNull(context.preferIO()); + Assertions.assertEquals("s3a://bucket/warehouse", context.options().get(CatalogOptions.WAREHOUSE)); + } + @Test public void filesystemSetsMetastoreFilesystemAndWarehouse() { Options opts = PaimonCatalogFactory.buildCatalogOptions( diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index 02d9049cc6ee97..a8f70e76066592 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -240,7 +240,11 @@ protected static void pingS3(String bucketName, String rootPath, Map properties, String provider) { // Validate the raw aliases before a preferred value can hide conflicting credentials. GcsAuthResolver.resolve(properties); - if (provider == null) { + if (StringUtils.isBlank(provider) && GcsAuthResolver.guessIsGcs(properties)) { + provider = "GCP"; + properties.put("provider", provider); + } + if (StringUtils.isBlank(provider)) { return; } switch (provider.toUpperCase(Locale.ROOT)) { @@ -267,6 +271,14 @@ public synchronized void modifyProperties(Map newProperties) thr Map properties = new HashMap<>(newProperties); String provider = StringUtils.defaultIfEmpty(properties.get("provider"), this.properties.get("provider")); + if (StringUtils.isBlank(provider)) { + Map selectionProperties = new HashMap<>(this.properties); + selectionProperties.putAll(properties); + if (GcsAuthResolver.guessIsGcs(selectionProperties)) { + provider = "GCP"; + properties.put("provider", provider); + } + } // Preserve AWS_* ALTER compatibility before merging with stored canonical properties. S3ResourceCompat.convertToStdProperties(properties); // Resolve aliases separately so this ALTER wins over persisted values regardless diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 5ae73b8adb1ba4..a7d1c0ba95e7f8 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -429,6 +429,83 @@ public void testGsEndpointAndAlterAliases() throws Exception { Assertions.assertEquals("https://old.example.com", properties.get("s3.endpoint")); } + @Test + public void testExplicitS3DefaultChainSurvivesGcsPolicyAndVaultCreation() throws Exception { + Map properties = new HashMap<>(); + properties.put("provider", "S3"); + properties.put("type", "s3"); + properties.put("s3.endpoint", "https://storage.googleapis.com"); + properties.put("s3.region", "us-central1"); + properties.put("s3.bucket", "bucket"); + properties.put("s3.root.path", "prefix"); + AtomicBoolean pinged = new AtomicBoolean(); + try (MockedStatic resourceMock = Mockito.mockStatic(S3Resource.class, Mockito.CALLS_REAL_METHODS); + MockedStatic envMock = Mockito.mockStatic(Env.class)) { + envMock.when(Env::getCurrentEnv).thenReturn(Mockito.mock(Env.class)); + resourceMock.when(() -> S3Resource.pingS3(Mockito.anyString(), Mockito.anyString(), Mockito.anyMap())) + .thenAnswer(invocation -> { + Map pingProperties = invocation.getArgument(2); + StorageAdapter adapter = StorageAdapter.of(pingProperties); + Assertions.assertEquals("S3", adapter.getSpiProperties().providerName()); + Assertions.assertEquals("DEFAULT", adapter.getSpiProperties().toMap() + .get("AWS_CREDENTIALS_PROVIDER_TYPE")); + pinged.set(true); + return null; + }); + CreateResourceCommand command = new CreateResourceCommand( + new CreateResourceInfo(false, false, "s3_vault", ImmutableMap.copyOf(properties))); + command.getInfo().analyzeResourceType(); + S3StorageVault vault = new S3StorageVault("s3_vault", false, false, command); + Assertions.assertTrue(pinged.get()); + Map stored = vault.getCopiedProperties(); + Assertions.assertFalse(S3ThriftAdapter.getS3TStorageParam(stored).isSetCredProviderType()); + Assertions.assertFalse(CloudObjectStoreAdapter.getObjStoreInfoPB(stored).hasCredProviderType()); + // Anonymous remains an explicit choice and remains forbidden for GCS vaults. + stored.put("s3.credentials_provider_type", "ANONYMOUS"); + Assertions.assertEquals(org.apache.doris.thrift.TCredProviderType.ANONYMOUS, + S3ThriftAdapter.getS3TStorageParam(stored).getCredProviderType()); + Assertions.assertThrows(IllegalArgumentException.class, + () -> CloudObjectStoreAdapter.getObjStoreInfoPB(stored)); + } + } + + @Test + public void testInferredGcpAliasesForResourceAndVault() throws Exception { + Map properties = new HashMap<>(); + properties.put("type", "s3"); + properties.put("gs.endpoint", "https://storage.googleapis.com"); + properties.put("gs.credential_provider_type", "DEFAULT"); + properties.put("s3.bucket", "bucket"); + properties.put("s3.root.path", "prefix"); + properties.put("s3_validity_check", "false"); + S3Resource resource = new S3Resource("inferred_gcp"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Assertions.assertEquals("GCP", resource.getProperty("provider")); + Assertions.assertEquals(properties.get("gs.endpoint"), resource.getProperty("s3.endpoint")); + resource.modifyProperties(ImmutableMap.of("gs.connection.timeout", "789")); + Assertions.assertEquals("789", resource.getProperty("s3.connection.timeout")); + try (MockedStatic envMock = Mockito.mockStatic(Env.class)) { + envMock.when(Env::getCurrentEnv).thenReturn(Mockito.mock(Env.class)); + CreateResourceCommand command = new CreateResourceCommand( + new CreateResourceInfo(false, false, "gcp_vault", ImmutableMap.copyOf(properties))); + command.getInfo().analyzeResourceType(); + S3StorageVault vault = new S3StorageVault("gcp_vault", false, false, command); + Assertions.assertEquals("https://storage.googleapis.com", + CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()).getEndpoint()); + Assertions.assertTrue(CloudObjectStoreAdapter.getObjStoreInfoPB(vault.getCopiedProperties()) + .getCredential().hasGcpCredential()); + } + // Legacy persisted resources can have an inferred identity without a provider field. + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + Map legacy = resource.getCopiedProperties(); + legacy.remove("provider"); + field.set(resource, legacy); + resource.modifyProperties(ImmutableMap.of("gs.connection.timeout", "987")); + Assertions.assertEquals("987", resource.getProperty("s3.connection.timeout")); + Assertions.assertEquals("GCP", resource.getProperty("provider")); + } + @Test public void testGsEndpointForStorageVault() throws Exception { Env env = Mockito.mock(Env.class); @@ -524,7 +601,7 @@ public void testGsEndpointCannotAlterPolicyResource() throws Exception { @Test public void testGsEndpointDoesNotOverrideOtherProviders() throws Exception { - for (String provider : new String[] {"S3", "OSS", "AZURE", ""}) { + for (String provider : new String[] {"S3", "OSS", "AZURE"}) { Map properties = new HashMap<>(s3Properties); if (!provider.isEmpty()) { properties.put("provider", provider); diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java index ba963ceb00b35f..cb01d70cde36cb 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java @@ -46,17 +46,16 @@ public static Optional resolve(Map properties) { boolean hasSecretKey = hasNonBlankProperty(properties, SECRET_KEYS); boolean hasLegacyAnonymous = hasLegacyAnonymousProvider(properties); String provider = getPropertyIgnoreCase(properties, "provider"); - boolean useLegacyAnonymousDefault = "S3".equalsIgnoreCase(provider) - && !hasNonBlankProperty(properties, AWS_PROVIDERS); if (isNotBlank(provider) && !"GCP".equalsIgnoreCase(provider) && !"GCS".equalsIgnoreCase(provider)) { if (hasNativeProperties) { throw new IllegalArgumentException("Native GCP authentication requires provider=GCP, but found: " + provider); } // Legacy GCS configurations may explicitly select the S3-compatible protocol. - // Preserve their HMAC/anonymous authentication without enabling native ADC for S3. + // Preserve explicit HMAC/anonymous authentication without enabling native ADC for S3. + // Omitted credentials still belong to S3's default chain, even at a GCS endpoint. if (!"S3".equalsIgnoreCase(provider) || !hasGcsSelector - || !(hasAccessKey || hasSecretKey || hasLegacyAnonymous || useLegacyAnonymousDefault)) { + || !(hasAccessKey || hasSecretKey || hasLegacyAnonymous)) { return Optional.empty(); } } @@ -76,8 +75,6 @@ public static Optional resolve(Map properties) { mode = GcsAuth.Mode.ANONYMOUS; } else if (hasAccessKey || hasSecretKey) { mode = GcsAuth.Mode.HMAC; - } else if (useLegacyAnonymousDefault) { - mode = GcsAuth.Mode.ANONYMOUS; } else { mode = GcsAuth.Mode.ADC; } diff --git a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java index b9ca61e6cc9b77..5e7bbfdbd0f80d 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java @@ -87,17 +87,17 @@ public void testLegacyS3HmacRequiresGcsSelector() { } @Test - public void testLegacyS3OmittedAuthenticationUsesAnonymousOnlyForGcs() { + public void testExplicitS3OmittedAuthenticationPreservesDefaultChain() { Map props = new HashMap<>(); props.put("provider", "s3"); Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); props.put("s3.endpoint", "https://storage.googleapis.com"); - GcsAuth auth = GcsAuthResolver.resolve(props).get(); - Assertions.assertEquals(GcsAuth.Mode.ANONYMOUS, auth.getMode()); - Assertions.assertFalse(auth.getNativeCredential().isPresent()); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); props.put("s3.credentials_provider_type", "DEFAULT"); Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); + props.put("s3.credentials_provider_type", "ANONYMOUS"); + Assertions.assertEquals(GcsAuth.Mode.ANONYMOUS, GcsAuthResolver.resolve(props).get().getMode()); props.remove("s3.credentials_provider_type"); props.put("provider", "GCP"); Assertions.assertEquals(GcsAuth.Mode.ADC, GcsAuthResolver.resolve(props).get().getMode()); @@ -111,11 +111,7 @@ public void testOtherProvidersAndConflictingNativeProperties() { Map props = new HashMap<>(); props.put("provider", provider); props.put("s3.endpoint", "storage.googleapis.com"); - if ("S3".equals(provider)) { - Assertions.assertEquals(GcsAuth.Mode.ANONYMOUS, GcsAuthResolver.resolve(props).get().getMode()); - } else { - Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); - } + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent()); props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index 5244fedd685af6..90a715749dbe68 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -139,10 +139,11 @@ public void validate() { GcpAuth.validateEndpoint(getEndpoint()); // Connectors overlay raw Hadoop properties after this canonical configuration. // Keep the OAuth destination on the validated gs.endpoint path. - for (String key : Set.of("fs.gs.storage.root.url", "fs.gs.storage.service.path")) { + for (String key : Set.of("fs.gs.storage.root.url", "fs.gs.storage.service.path", + "fs.gs.token.server.url")) { if (rawProperties().containsKey(key)) { throw new IllegalArgumentException("Native GCP authentication does not allow " + key - + "; configure the storage endpoint with gs.endpoint instead"); + + "; use the native GCP credential and endpoint properties instead"); } } // Raw auth overrides would make Hadoop use a different identity from object clients. diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index 5db5b350d71203..e5ee88846c6a1e 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -248,7 +248,8 @@ void nativeModesPropagateToBackendHadoopAndCache() { @Test void nativeHadoopEndpointCannotBeOverriddenByRawProperties() { for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { - for (String key : new String[] {"fs.gs.storage.root.url", "fs.gs.storage.service.path"}) { + for (String key : new String[] {"fs.gs.storage.root.url", "fs.gs.storage.service.path", + "fs.gs.token.server.url"}) { IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, () -> GcsFileSystemProperties.of(Map.of("provider", "GCP", "gs.credential_provider_type", mode, key, "https://untrusted.example/"))); diff --git a/fe/fe-paimon-common/pom.xml b/fe/fe-paimon-common/pom.xml new file mode 100644 index 00000000000000..52dd2132e06390 --- /dev/null +++ b/fe/fe-paimon-common/pom.xml @@ -0,0 +1,50 @@ + + + + 4.0.0 + + org.apache.doris + fe + ${revision} + ../pom.xml + + fe-paimon-common + Doris Shared Paimon FileIO + GCS URI normalization shared by FE planning and serialized BE Paimon readers. + + + + org.apache.paimon + paimon-common + provided + + + org.apache.hadoop + hadoop-common + provided + + + org.junit.jupiter + junit-jupiter + test + + + diff --git a/fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java b/fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java new file mode 100644 index 00000000000000..58a2db2349aff9 --- /dev/null +++ b/fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java @@ -0,0 +1,117 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.paimon; + +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.FileIOLoader; +import org.apache.paimon.fs.FileStatus; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.PositionOutputStream; +import org.apache.paimon.fs.RemoteIterator; +import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.fs.hadoop.HadoopFileIO; + +import java.io.IOException; + +/** Routes legacy GCS aliases through the native Hadoop GCS connector, including persisted file paths. */ +public class NativeGcsFileIO extends HadoopFileIO { + private static final long serialVersionUID = 1L; + + public static Path normalize(Path path) { + String scheme = path.toUri().getScheme(); + if ("s3".equalsIgnoreCase(scheme) || "s3a".equalsIgnoreCase(scheme)) { + return new Path("gs" + path.toString().substring(scheme.length())); + } + return path; + } + + @Override + public boolean isObjectStore() { + return true; + } + + @Override + public SeekableInputStream newInputStream(Path path) throws IOException { + return super.newInputStream(normalize(path)); + } + + @Override + public PositionOutputStream newOutputStream(Path path, boolean overwrite) throws IOException { + return super.newOutputStream(normalize(path), overwrite); + } + + @Override + public FileStatus getFileStatus(Path path) throws IOException { + return super.getFileStatus(normalize(path)); + } + + @Override + public FileStatus[] listStatus(Path path) throws IOException { + return super.listStatus(normalize(path)); + } + + @Override + public RemoteIterator listFilesIterative(Path path, boolean recursive) throws IOException { + return super.listFilesIterative(normalize(path), recursive); + } + + @Override + public boolean exists(Path path) throws IOException { + return super.exists(normalize(path)); + } + + @Override + public boolean delete(Path path, boolean recursive) throws IOException { + return super.delete(normalize(path), recursive); + } + + @Override + public boolean mkdirs(Path path) throws IOException { + return super.mkdirs(normalize(path)); + } + + @Override + public boolean rename(Path source, Path target) throws IOException { + return super.rename(normalize(source), normalize(target)); + } + + @Override + public void overwriteFileUtf8(Path path, String content) throws IOException { + super.overwriteFileUtf8(normalize(path), content); + } + + @Override + public boolean tryAtomicOverwriteViaRename(Path path, String content) throws IOException { + return super.tryAtomicOverwriteViaRename(normalize(path), content); + } + + public static final class Loader implements FileIOLoader { + private static final long serialVersionUID = 1L; + + @Override + public String getScheme() { + return "gs"; + } + + @Override + public FileIO load(Path path) { + return "gs".equalsIgnoreCase(normalize(path).toUri().getScheme()) + ? new NativeGcsFileIO() : new HadoopFileIO(); + } + } +} diff --git a/fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java b/fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java new file mode 100644 index 00000000000000..37d28da660fe85 --- /dev/null +++ b/fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java @@ -0,0 +1,106 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.paimon; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.BlockLocation; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.RawLocalFileSystem; +import org.apache.paimon.catalog.CatalogContext; +import org.apache.paimon.fs.Path; +import org.apache.paimon.options.Options; +import org.apache.paimon.utils.InstantiationUtil; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.File; +import java.io.IOException; +import java.net.URI; + +public class NativeGcsFileIOTest { + @TempDir + public java.nio.file.Path directory; + + @Test + public void aliasesUseNativeFilesystemBeforeAndAfterSerialization() throws Exception { + Configuration conf = new Configuration(false); + conf.set("fs.gs.impl", LocalGcsFileSystem.class.getName()); + conf.setBoolean("fs.gs.impl.disable.cache", true); + conf.set("test.gcs.root", directory.toString()); + conf.set("fs.gs.auth.type", "APPLICATION_DEFAULT"); + NativeGcsFileIO original = new NativeGcsFileIO(); + original.configure(CatalogContext.create(new Options(), conf)); + NativeGcsFileIO restored = InstantiationUtil.deserializeObject( + InstantiationUtil.serializeObject(original), getClass().getClassLoader()); + for (NativeGcsFileIO fileIO : new NativeGcsFileIO[] {original, restored}) { + Assertions.assertEquals("APPLICATION_DEFAULT", fileIO.hadoopConf().get("fs.gs.auth.type")); + Assertions.assertTrue(fileIO.isObjectStore()); + for (String scheme : new String[] {"s3", "s3a", "gs"}) { + Path folder = new Path(scheme + "://bucket/" + scheme); + Path file = new Path(folder, "manifest"); + Assertions.assertTrue(fileIO.mkdirs(folder)); + fileIO.writeFile(file, "private-gcs-data", true); + Assertions.assertEquals("private-gcs-data", fileIO.readFileUtf8(file)); + Assertions.assertTrue(fileIO.exists(file)); + Assertions.assertEquals(16, fileIO.getFileStatus(file).getLen()); + Assertions.assertEquals(1, fileIO.listStatus(folder).length); + Assertions.assertTrue(fileIO.listFilesIterative(folder, true).hasNext()); + Path target = new Path("s3a://bucket/" + scheme + "/renamed"); + Assertions.assertTrue(fileIO.rename(file, target)); + Assertions.assertTrue(fileIO.delete(target, false)); + } + } + } + + @Test + public void normalizationPreservesObjectKeyAndOtherSchemes() { + Assertions.assertEquals("gs://bucket/a%20b/part-1", + NativeGcsFileIO.normalize(new Path("s3a://bucket/a%20b/part-1")).toString()); + for (String uri : new String[] {"gs://bucket/key", "hdfs://namenode/key", "file:/tmp/key"}) { + Path path = new Path(uri); + Assertions.assertSame(path, NativeGcsFileIO.normalize(path)); + } + } + + /** Local backing storage that rejects any path not normalized to gs before Hadoop dispatch. */ + public static class LocalGcsFileSystem extends RawLocalFileSystem { + @Override + public URI getUri() { + return URI.create("gs://bucket"); + } + + @Override + public FileStatus getFileStatus(org.apache.hadoop.fs.Path path) throws IOException { + FileStatus status = super.getFileStatus(path); + return new FileStatus(status.getLen(), status.isDirectory(), 1, 4096, + status.getModificationTime(), path); + } + + @Override + public BlockLocation[] getFileBlockLocations(FileStatus status, long start, long length) { + return new BlockLocation[0]; + } + + @Override + public File pathToFile(org.apache.hadoop.fs.Path path) { + Assertions.assertEquals("gs", path.toUri().getScheme()); + return new File(getConf().get("test.gcs.root"), path.toUri().getPath()); + } + } +} diff --git a/fe/pom.xml b/fe/pom.xml index 41b2c314a4d915..e27571edfbd415 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -231,6 +231,7 @@ under the License. fe-common fe-trino-connector-common fe-iceberg-common + fe-paimon-common fe-catalog fe-filesystem fe-extension-spi From 9bf90ec08a5954a26c96b9dcc77776d50d9be298 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 16:26:26 +0800 Subject: [PATCH 17/47] [fix](test) Use the storage adapter backend properties API ### What problem does this PR solve? Related PR: #68540 Problem Summary: TeamCity Compile build 1064419 fails during fe-core testCompile because S3ResourceTest calls toMap() on FileSystemProperties, which does not declare that method. Read the authentication mode through StorageAdapter.getBackendConfigProperties(), preserving the assertion that explicit S3 storage at a GCS endpoint uses DEFAULT authentication. ### Release note None ### Check List (For Author) - Test: fe-core Checkstyle passes with zero violations; git diff --check passes. Verified the replacement method signature and the DEFAULT mode mapping. No remaining getSpiProperties().toMap() calls in Java sources. run-fe-ut.sh -t S3ResourceTest stops before compilation because the local JDK is 20 and the script requires JDK 17; full validation remains for CI. - Behavior changed: No; test compilation fix only. - Does this need documentation: No. --- .../src/test/java/org/apache/doris/catalog/S3ResourceTest.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index a7d1c0ba95e7f8..83a702a6691770 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -447,7 +447,7 @@ public void testExplicitS3DefaultChainSurvivesGcsPolicyAndVaultCreation() throws Map pingProperties = invocation.getArgument(2); StorageAdapter adapter = StorageAdapter.of(pingProperties); Assertions.assertEquals("S3", adapter.getSpiProperties().providerName()); - Assertions.assertEquals("DEFAULT", adapter.getSpiProperties().toMap() + Assertions.assertEquals("DEFAULT", adapter.getBackendConfigProperties() .get("AWS_CREDENTIALS_PROVIDER_TYPE")); pinged.set(true); return null; From f479d053562b076705cc990ed07a9ef26e2db395 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 16:48:37 +0800 Subject: [PATCH 18/47] [fix](fe) Allow resource ALTER to clear GCP impersonation ### What problem does this PR solve? Related PR: #68540 Problem Summary: ALTER RESOURCE ignored an explicitly empty gs.impersonation_service_account and continued validating and publishing the previous impersonated identity. Preserve this explicit empty update in the effective properties so ping, persistence, and storage-policy credentials use the source identity. Omitted impersonation properties continue preserving the configured account. ### Release note Fix clearing GCP impersonation through ALTER RESOURCE. ### Check List (For Author) - Test: Added DEFAULT and COMPUTE_ENGINE regression coverage for omitted and empty impersonation, ping credentials, serialization and Thrift output, and unchanged properties/version after a failed ping. fe-core Checkstyle and git diff --check pass. The S3ResourceTest runner stops before compilation because JDK 17 is required and the local usable JDK is 20; the new tests remain unexecuted locally. - Behavior changed: Explicit empty impersonation clears the stored account. - Does this need documentation: No. --- .../org/apache/doris/catalog/S3Resource.java | 6 +- .../apache/doris/catalog/S3ResourceTest.java | 96 +++++++++++++++++++ 2 files changed, 100 insertions(+), 2 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index a8f70e76066592..03ce805b250f8b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -24,6 +24,7 @@ import org.apache.doris.common.util.S3Util; import org.apache.doris.datasource.storage.S3ResourceCompat; import org.apache.doris.filesystem.UploadPartResult; +import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.filesystem.spi.ObjStorage; @@ -288,10 +289,11 @@ public synchronized void modifyProperties(Map newProperties) thr normalizeProperties(effectiveProperties, provider); S3ResourceCompat.convertToStdProperties(effectiveProperties); for (Map.Entry update : properties.entrySet()) { - // Match persistence: empty updates are ignored, except when clearing a session token. + // Empty updates are ignored, except when clearing a session token or impersonation account. replaceIfEffectiveValue(effectiveProperties, update.getKey(), update.getValue()); if (S3ResourceCompat.SESSION_TOKEN.equals(update.getKey()) - || S3ResourceCompat.Env.TOKEN.equals(update.getKey())) { + || S3ResourceCompat.Env.TOKEN.equals(update.getKey()) + || GcpCredential.IMPERSONATION_SERVICE_ACCOUNT.equals(update.getKey())) { effectiveProperties.put(update.getKey(), update.getValue()); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 83a702a6691770..57182bcdd37819 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -33,6 +33,7 @@ import org.apache.doris.nereids.trees.plans.commands.CreateResourceCommand; import org.apache.doris.nereids.trees.plans.commands.info.CreateResourceInfo; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.thrift.TGcpCredential; import org.apache.doris.thrift.TS3StorageParam; import com.google.common.base.Strings; @@ -46,6 +47,8 @@ import org.mockito.MockedStatic; import org.mockito.Mockito; +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; import java.io.DataInputStream; import java.io.DataOutputStream; import java.lang.reflect.Field; @@ -732,6 +735,99 @@ public void testGsAliasesDoNotOverrideOtherProviders() throws Exception { } } + @Test + public void testAlterClearsGcpImpersonation() throws Exception { + MetaContext previousContext = MetaContext.get(); + MetaContext metaContext = new MetaContext(); + metaContext.setMetaVersion(FeMetaVersion.VERSION_CURRENT); + metaContext.setThreadLocalInfo(); + try { + for (String providerType : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + try (MockedStatic resourceMock = Mockito.mockStatic( + S3Resource.class, Mockito.CALLS_REAL_METHODS)) { + S3Resource resource = createGcpResourceWithImpersonation(providerType); + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.CONNECTION_TIMEOUT_MS, "2000")); + Assertions.assertEquals("reader@project.iam.gserviceaccount.com", + resource.getProperty(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + AtomicBoolean pinged = new AtomicBoolean(); + resourceMock.when(() -> S3Resource.pingS3(Mockito.anyString(), Mockito.anyString(), Mockito.anyMap())) + .thenAnswer(invocation -> { + Map pingProperties = invocation.getArgument(2); + Assertions.assertEquals("", pingProperties.get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + TGcpCredential credential = S3ThriftAdapter.getS3TStorageParam(pingProperties) + .getCredential().getGcpCredential(); + Assertions.assertEquals(providerType, credential.getCredentialProviderType().name()); + Assertions.assertFalse(credential.isSetImpersonationServiceAccount()); + pinged.set(true); + return null; + }); + long previousVersion = resource.getVersion(); + resource.modifyProperties(ImmutableMap.of(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "", + S3ResourceCompat.VALIDITY_CHECK, "true")); + Assertions.assertTrue(pinged.get()); + Assertions.assertEquals(previousVersion + 1, resource.getVersion()); + Assertions.assertEquals("", resource.getProperty(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (DataOutputStream output = new DataOutputStream(bytes)) { + resource.write(output); + } + try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + S3Resource restored = (S3Resource) Resource.read(input); + Assertions.assertEquals(resource.getCopiedProperties(), restored.getCopiedProperties()); + TGcpCredential credential = S3ThriftAdapter.getS3TStorageParam(restored.getCopiedProperties()) + .getCredential().getGcpCredential(); + Assertions.assertEquals(providerType, credential.getCredentialProviderType().name()); + Assertions.assertFalse(credential.isSetImpersonationServiceAccount()); + } + } + } + } finally { + if (previousContext == null) { + MetaContext.remove(); + } else { + previousContext.setThreadLocalInfo(); + } + } + } + + @Test + public void testFailedAlterPreservesGcpImpersonation() throws Exception { + try (MockedStatic resourceMock = Mockito.mockStatic(S3Resource.class, Mockito.CALLS_REAL_METHODS)) { + resourceMock.when(() -> S3Resource.pingS3(Mockito.anyString(), Mockito.anyString(), Mockito.anyMap())) + .thenAnswer(invocation -> { + Map pingProperties = invocation.getArgument(2); + Assertions.assertEquals("", pingProperties.get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + throw new DdlException("source credential cannot access bucket"); + }); + for (String providerType : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + S3Resource resource = createGcpResourceWithImpersonation(providerType); + Map previousProperties = resource.getCopiedProperties(); + long previousVersion = resource.getVersion(); + Assertions.assertThrows(DdlException.class, () -> resource.modifyProperties( + ImmutableMap.of(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "", + S3ResourceCompat.VALIDITY_CHECK, "true"))); + Assertions.assertEquals(previousProperties, resource.getCopiedProperties()); + Assertions.assertEquals(previousVersion, resource.getVersion()); + } + } + } + + private S3Resource createGcpResourceWithImpersonation(String providerType) throws DdlException { + Map properties = new HashMap<>(); + properties.put("provider", "GCP"); + properties.put(S3ResourceCompat.ENDPOINT, "https://storage.googleapis.com"); + properties.put(S3ResourceCompat.REGION, "us-central1"); + properties.put(S3ResourceCompat.BUCKET, "bucket"); + properties.put(S3ResourceCompat.ROOT_PATH, "prefix"); + properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, providerType); + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "reader@project.iam.gserviceaccount.com"); + S3Resource resource = new S3Resource("gcp_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + return resource; + } + @Test public void testGsCredentialAliasesCannotBypassNativeAuthValidation() throws Exception { Map properties = new HashMap<>(); From b60f30a4e1c67a5370d07112073d8257af246832 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 17:28:20 +0800 Subject: [PATCH 19/47] [fix](build) Scope optional GCS plugin closure dependencies ### What problem does this PR solve? Related PR: #68540 Problem Summary: TeamCity Compile 1064581 completes Maven compilation but fails the deployed plugin closure check. Rooting the shaded GCS connector exposes optional Netty codecs and TLS helpers, App Engine transport, native-image and test helpers, compile-time annotations, and optional OpenTelemetry incubator classes. GcsDtFetcher also contains an unused pre-shading class constant. Keep the GCS jar in the root set and allow only documented source/target pairs in the Hudi, Iceberg and Paimon plugins. Required filesystem and authentication dependencies, including references to the same targets from Doris code, remain checked. ### Release note None ### Check List (For Author) - Test: Five Python regression tests pass, including red/green exception removal, missing runtime dependencies, source and plugin scoping, and jdeps failure handling. All 317 CI diagnostics and their 825 actual bytecode reference edges match the scoped rules. A local runtime smoke test exercises the shaded telemetry SDK without incubator classes. git diff --check passes. Full deployed-plugin build remains for CI. - Behavior changed: Optional GCS jar references no longer fail packaging; required dependency checks and deployed jars are unchanged. - Does this need documentation: No; each exception records its reason. --- tools/be-java-plugins/check_plugin_layout.py | 54 ++++++++ .../test_check_plugin_layout.py | 122 ++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 tools/be-java-plugins/test_check_plugin_layout.py diff --git a/tools/be-java-plugins/check_plugin_layout.py b/tools/be-java-plugins/check_plugin_layout.py index e63510e01f8a91..7d489a9152b59a 100755 --- a/tools/be-java-plugins/check_plugin_layout.py +++ b/tools/be-java-plugins/check_plugin_layout.py @@ -216,6 +216,58 @@ } +# gcs-connector 3.1.18 is a shaded uber-jar. Making its Hadoop filesystem a root +# also scans optional transports, test helpers and native-image substitutions. +# Match BOTH ends of these edges: a missing library referenced by Doris or the +# GCS filesystem/authentication implementation must still fail the check. +_GCS = "com.google.cloud.hadoop.repackaged.gcs." +_GCS_NETTY = _GCS + "io.grpc.netty.shaded.io.netty." +_GCS_CODECS = ("Optional Netty codecs bundled in the GCS jar. GCS uses HTTP/gRPC, not " + "SPDY, JBoss marshalling or protobuf-nano. Its gzip/deflate path uses " + "JDK zlib; Brotli and Zstd HTTP handlers probe isAvailable() before use.") +_GCS_ANNOTATIONS = ("Compile-time annotations in the shaded GCS jar; the generated " + "AutoValue implementations are already bundled.") +# Each tuple is (referencing class prefix, missing class prefix, reason). +GCS_CLOSURE_ALLOWLIST = [ + (_GCS + "com.google.api.client.extensions.appengine.http.", "com.google.appengine.api.urlfetch.", + "App Engine's UrlFetch transport; BE uses the normal JVM HTTP transport."), + ("com.google.cloud.hadoop.", "com.google.auto.value.AutoValue", _GCS_ANNOTATIONS), + (_GCS + "com.google.common.flogger.", "org.checkerframework.checker.nullness.compatqual.", + _GCS_ANNOTATIONS), + (_GCS_NETTY + "util.concurrent.", "org.jetbrains.annotations.Async", _GCS_ANNOTATIONS), + (_GCS_NETTY + "util.", "com.oracle.svm.core.annotate.", + "Graal native-image substitutions are not used by BE's JVM."), + (_GCS_NETTY + "util.internal.Hidden$NettyBlockHoundIntegration", "reactor.blockhound.", + "Optional BlockHound integration; BE does not install a BlockHound agent."), + (_GCS_NETTY + "handler.ssl.", "org.bouncycastle.", + "Optional PEM/self-signed certificate helpers. GCS uses trust-store TLS; " + "BouncyCastlePemReader checks availability and JDK PEM parsing is the fallback."), + (_GCS_NETTY + "handler.ssl.JettyAlpnSslEngine", "org.eclipse.jetty.alpn.", + "Legacy Jetty ALPN for old JDKs; BE requires JDK 17 and uses JDK ALPN."), + (_GCS_NETTY + "handler.ssl.JettyNpnSslEngine", "org.eclipse.jetty.npn.", + "Legacy NPN transport; GCS uses HTTP/2 ALPN on JDK 17."), + (_GCS + "io.grpc.testing.", "org.junit.", + "Upstream GrpcCleanupRule/GrpcServerRule test helpers bundled in the shaded runtime jar."), + (_GCS + "io.opentelemetry.sdk.", _GCS + "io.opentelemetry.api.incubator.", + "SdkTracer, SdkLogger and instrument builders probe incubator availability " + "with Class.forName and use the stable implementation when it is absent."), + ("com.google.cloud.hadoop.fs.gcs.auth.GcsDtFetcher", + "com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem", + "Unused pre-shading constant-pool entry in 3.1.18: javap shows no instruction " + "or signature using it. The actual filesystem implementation is relocated."), +] + [(_GCS_NETTY + "handler.codec.", target, _GCS_CODECS) for target in ( + "com.aayushatharva.brotli4j.", "com.github.luben.zstd.", "com.jcraft.jzlib.", + "com.ning.compress.", "lzma.sdk.", "net.jpountz.lz4.", "net.jpountz.xxhash.", + "org.jboss.marshalling.", _GCS + "com.google.protobuf.nano.", +)] + + +def _gcs_optional_reference(plugin, source, target): + return plugin in ("hudi", "iceberg", "paimon") and any( + source.startswith(src) and target.startswith(dst) + for src, dst, _reason in GCS_CLOSURE_ALLOWLIST) + + def _allowed(rules, path): for kind, pattern, _reason in rules: if kind == "basename" and path.rsplit("/", 1)[-1] == pattern: @@ -518,6 +570,8 @@ def check_closure_self_contained(plugin, jars, spi_jar, fail): continue if any(target.startswith(prefix) for prefix, _reason in rules): continue + if _gcs_optional_reference(plugin, src, target): + continue missing[target].add(src) for target in sorted(missing): fail("closure-self-contained", diff --git a/tools/be-java-plugins/test_check_plugin_layout.py b/tools/be-java-plugins/test_check_plugin_layout.py new file mode 100644 index 00000000000000..50a0199819a5d4 --- /dev/null +++ b/tools/be-java-plugins/test_check_plugin_layout.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Regression tests for source-scoped GCS dependency exceptions. + +Run with: python3 -m unittest discover -s tools/be-java-plugins -p 'test_*.py' +""" +import os +import subprocess +import tempfile +import unittest +from unittest import mock +import zipfile + +import check_plugin_layout as layout + + +class GcsClosureTest(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.addCleanup(self.directory.cleanup) + self.doris_jar = self.jar("scanner.jar", ["org/apache/doris/Scanner.class"]) + self.gcs_jar = self.jar("gcs.jar", [ + "com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystem.class"]) + self.spi_jar = self.jar("spi.jar", []) + + def jar(self, name, classes): + path = os.path.join(self.directory.name, name) + with zipfile.ZipFile(path, "w") as jar: + for entry in classes: + jar.writestr(entry, b"fixture") + return path + + def check_edges(self, plugin, edges): + output = "\n".join(" %s -> %s not found" % edge for edge in edges) + failures = [] + with mock.patch.object(layout.subprocess, "run", return_value=subprocess.CompletedProcess( + [], 0, stdout=output, stderr="")) as run: + layout.check_closure_self_contained( + plugin, [self.doris_jar, self.gcs_jar], self.spi_jar, + lambda check, message: failures.append((check, message))) + # The GCS jar remains a jdeps root; the fix must not hide the whole jar. + self.assertIn(self.gcs_jar, run.call_args.args[0]) + return failures + + def test_optional_gcs_edges(self): + edges = [ + (layout._GCS + "com.google.api.client.extensions.appengine.http.UrlFetchRequest", + "com.google.appengine.api.urlfetch.HTTPRequest"), + (layout._GCS_NETTY + "handler.codec.compression.BrotliDecoder", + "com.aayushatharva.brotli4j.decoder.DecoderJNI"), + (layout._GCS_NETTY + "handler.ssl.BouncyCastlePemReader", + "org.bouncycastle.openssl.PEMParser"), + (layout._GCS + "io.opentelemetry.sdk.trace.ExtendedSdkTracer", + layout._GCS + "io.opentelemetry.api.incubator.trace.ExtendedTracer"), + (layout._GCS + "io.grpc.testing.GrpcCleanupRule", "org.junit.runners.model.Statement"), + (layout._GCS + "io.grpc.testing.GrpcServerRule", "org.junit.rules.ExternalResource"), + ("com.google.cloud.hadoop.fs.gcs.auth.GcsDtFetcher", + "com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem"), + ] + for plugin in ("hudi", "iceberg", "paimon"): + with self.subTest(plugin=plugin): + self.assertEqual([], self.check_edges(plugin, edges)) + with mock.patch.object(layout, "GCS_CLOSURE_ALLOWLIST", []): + self.assertEqual(len(edges), len(self.check_edges(plugin, edges))) + + def test_same_target_from_runtime_code_still_fails(self): + target = "org.bouncycastle.openssl.PEMParser" + edges = [ + (layout._GCS_NETTY + "handler.ssl.BouncyCastlePemReader", target), + ("org.apache.doris.GcsReader", target), + ] + for plugin in ("hudi", "iceberg", "paimon"): + failures = self.check_edges(plugin, edges) + self.assertEqual(1, len(failures)) + self.assertIn("org.apache.doris.GcsReader", failures[0][1]) + self.assertIn("Either ship the jar", failures[0][1]) + + def test_required_gcs_dependencies_still_fail(self): + edges = [ + ("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + "org.apache.hadoop.fs.FileSystem"), + (layout._GCS + "com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl", + layout._GCS + "com.google.auth.oauth2.GoogleCredentials"), + (layout._GCS + "com.google.auth.oauth2.ServiceAccountCredentials", + "org.bouncycastle.openssl.PEMParser"), + (layout._GCS_NETTY + "handler.ssl.JdkSslContext", "missing.RequiredTlsClass"), + ] + for plugin in ("hudi", "iceberg", "paimon"): + self.assertEqual(len(edges), len(self.check_edges(plugin, edges))) + + def test_exceptions_do_not_apply_to_other_plugins(self): + edges = [(layout._GCS + "io.grpc.testing.GrpcCleanupRule", "org.junit.rules.ExternalResource")] + self.assertEqual(1, len(self.check_edges("jdbc", edges))) + + def test_jdeps_failure_is_not_ignored(self): + failures = [] + with mock.patch.object(layout.subprocess, "run", return_value=subprocess.CompletedProcess( + [], 1, stdout="", stderr="broken classpath")): + layout.check_closure_self_contained( + "paimon", [self.doris_jar, self.gcs_jar], self.spi_jar, + lambda check, message: failures.append(message)) + self.assertEqual(1, len(failures)) + self.assertIn("jdeps exited 1", failures[0]) + + +if __name__ == "__main__": + unittest.main() From d4f4a8ddbec75f162c46655fec74958ac88bd8c4 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 20:20:52 +0800 Subject: [PATCH 20/47] [fix](storage) Preserve GCP configuration on policy and Paimon REST paths ### What problem does this PR solve? Related PR: #68540 Problem Summary: BE policy resource resets copied native credentials while retaining the old provider, rejecting accepted S3-to-GCP updates. Copy the provider along with the credentials and cover S3-to-GCP-to-S3 client resets. Paimon REST catalogs used an options-only context, losing configured GCS authentication and the native alias FileIO loader for local storage paths. Build the REST context with the same canonical Hadoop configuration and FileIO loader as the other catalog flavors. Add a loopback REST fixture covering impersonation propagation and enabled/disabled data-token modes. ### Release note Fix native GCP provider changes on storage policies and GCS authentication configuration in Paimon REST catalogs. ### Check List (For Author) - Test: Added BE reset and Paimon REST catalog regression tests. Paimon Checkstyle, clang-format 16, connector import and metadata-funnel gates, BE build hygiene, and git diff --check pass. A standalone Paimon 1.4.2 SDK smoke check passes REST configuration round-trip, GCS context/loader, and both data-token modes without accessing GCS. Full regression tests remain unexecuted locally: the BE test runner requires JDK 17 (available runtime is 20), and FE generation lacks thirdparty protoc/thrift. The SDK smoke check is not a full connector test. - Behavior changed: Policy resets update provider; REST local FileIO keeps GCS credentials and URI alias support while retaining token selection. - Does this need documentation: No. --- be/src/io/fs/s3_file_system.cpp | 1 + be/test/io/s3_client_factory_test.cpp | 26 ++++++++ .../connector/paimon/PaimonConnector.java | 15 +++-- .../paimon/PaimonCatalogFactoryTest.java | 65 +++++++++++++++++++ 4 files changed, 100 insertions(+), 7 deletions(-) diff --git a/be/src/io/fs/s3_file_system.cpp b/be/src/io/fs/s3_file_system.cpp index 75298d6a65e3a1..1635c166617d5a 100644 --- a/be/src/io/fs/s3_file_system.cpp +++ b/be/src/io/fs/s3_file_system.cpp @@ -85,6 +85,7 @@ Status ObjClientHolder::reset(const S3ClientConf& conf) { { std::shared_lock lock(_mtx); reset_conf = _conf; + reset_conf.provider = conf.provider; reset_conf.ak = conf.ak; reset_conf.sk = conf.sk; reset_conf.token = conf.token; diff --git a/be/test/io/s3_client_factory_test.cpp b/be/test/io/s3_client_factory_test.cpp index 63524af7a44690..936564816dab5f 100644 --- a/be/test/io/s3_client_factory_test.cpp +++ b/be/test/io/s3_client_factory_test.cpp @@ -374,6 +374,32 @@ TEST_F(S3ClientFactoryTest, ObjClientHolderResetRefreshesNativeCredential) { EXPECT_EQ(holder.s3_client_conf(), rotated_conf); } +TEST_F(S3ClientFactoryTest, ObjClientHolderResetSwitchesS3AndGcpProviders) { + auto native_conf = make_gcp_native_conf(); + auto s3_conf = native_conf; + s3_conf.provider = io::ObjStorageProvider::AWS; + s3_conf.credential = std::monostate {}; + + std::vector created_confs; + auto client = std::make_shared(std::shared_ptr {}); + S3ClientFactory::instance().set_client_creator_for_test( + [&](const S3ClientConf& conf) -> std::shared_ptr { + created_confs.push_back(conf); + return client; + }); + + io::ObjClientHolder holder(s3_conf); + ASSERT_TRUE(holder.init().ok()); + // Existing policy resources take this reset path after receiving the new FE version. + ASSERT_TRUE(holder.reset(native_conf).ok()); + EXPECT_EQ(holder.s3_client_conf(), native_conf); + ASSERT_TRUE(holder.reset(s3_conf).ok()); + EXPECT_EQ(holder.s3_client_conf(), s3_conf); + ASSERT_EQ(created_confs.size(), 3); + EXPECT_EQ(created_confs[1], native_conf); + EXPECT_EQ(created_confs[2], s3_conf); +} + TEST_F(S3ClientFactoryTest, ObjClientHolderResetSwitchesNativeAndStaticCredentials) { auto native_conf = make_gcp_native_conf(); auto static_conf = native_conf; diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java index 097c15bc9d23a1..1646f2b650f024 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java @@ -428,9 +428,8 @@ private Catalog createCatalog() { // Canonical storage config from the FE-bound fe-filesystem StorageProperties (P1-T03), replacing // the legacy buildObjectStorageHadoopConfig path: object stores contribute their fs.s3a.*/fs.oss.* // /fs.cosn.*/fs.obs.* translation, and an HDFS-backed catalog contributes its hadoop.config.resources - // XML + HA + auth keys (C2; the defaults-free fe-filesystem Hadoop map). Empty for REST (the server - // owns storage) and for a catalog with no typed storage at all (it reaches the conf via the raw - // fs./dfs./hadoop. passthrough). + // XML + HA + auth keys (C2; the defaults-free fe-filesystem Hadoop map). Empty for a catalog + // with no typed storage at all (it reaches the conf via the raw fs./dfs./hadoop. passthrough). Map storageHadoopConfig = buildStorageHadoopConfig(); switch (flavor) { @@ -442,8 +441,10 @@ private Catalog createCatalog() { "Failed to create Paimon catalog with filesystem metastore"); } case PaimonCatalogProperties.REST: { - // rest is Options-only (no storage Configuration; the REST server owns storage). - return createCatalogFromContext(CatalogContext.create(options), flavor, + // External table paths and reads without REST data tokens use local FileIO. + Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration( + catalogProps.getRaw(), storageHadoopConfig); + return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, conf), flavor, "Failed to create Paimon catalog with REST metastore"); } case PaimonCatalogProperties.JDBC: { @@ -524,8 +525,8 @@ Options buildCatalogOptions() { * defaults-free so it never clobbers a co-bound object-store provider's tuned fs.s3a.* here). This * replaces the legacy {@code StorageProperties.buildObjectStorageHadoopConfig(properties)} call that * {@link PaimonCatalogFactory#buildHadoopConfiguration}/{@code buildHmsHiveConf} - * used to make. Empty for REST (the server owns storage) and for a catalog with no typed storage (it - * reaches the conf via the raw fs./dfs./hadoop. passthrough). + * used to make. Empty for a catalog with no typed storage (it reaches the conf via the raw + * fs./dfs./hadoop. passthrough). */ // Package-private (not private) so PaimonCatalogFactoryTest can drive the storage().getStorageProperties() // -> toHadoopProperties() -> Configuration wiring end-to-end (visible for testing). diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java index 87a9e1ab788cda..22776b364190f7 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java @@ -23,17 +23,25 @@ import org.apache.doris.filesystem.properties.StorageProperties; import org.apache.doris.paimon.NativeGcsFileIO; +import com.sun.net.httpserver.HttpServer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.paimon.catalog.Catalog; import org.apache.paimon.catalog.CatalogContext; +import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.fs.ResolvingFileIO; import org.apache.paimon.options.CatalogOptions; import org.apache.paimon.options.Options; +import org.apache.paimon.rest.RESTCatalog; +import org.apache.paimon.rest.RESTTokenFileIO; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import java.lang.reflect.Method; +import java.net.InetSocketAddress; import java.net.URL; import java.net.URLClassLoader; +import java.nio.charset.StandardCharsets; import java.util.Collections; import java.util.HashMap; import java.util.Map; @@ -105,6 +113,63 @@ public void nativeGcsNormalizesWarehouseAndInstallsAliasFileIO() { Assertions.assertEquals("s3a://bucket/warehouse", context.options().get(CatalogOptions.WAREHOUSE)); } + @Test + public void restCatalogRetainsNativeGcsContextAndTokenSelection() throws Exception { + HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext("/v1/config", exchange -> { + byte[] response = "{\"defaults\":{},\"overrides\":{}}".getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().set("Content-Type", "application/json"); + exchange.sendResponseHeaders(200, response.length); + try (java.io.OutputStream output = exchange.getResponseBody()) { + output.write(response); + } + }); + server.start(); + try { + for (boolean dataTokenEnabled : new boolean[] {false, true}) { + Map properties = props( + "paimon.catalog.type", "rest", + "paimon.rest.uri", "http://127.0.0.1:" + server.getAddress().getPort(), + "paimon.rest.prefix", "test-prefix", + "paimon.rest.token.provider", "bear", + "paimon.rest.token", "test-token", + "paimon.rest.data-token.enabled", Boolean.toString(dataTokenEnabled)); + Map gcsConfig = storage( + "fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + "fs.gs.auth.type", "APPLICATION_DEFAULT", + "fs.gs.auth.impersonation.service.account", "reader@project.iam.gserviceaccount.com"); + RecordingConnectorContext engineContext = new RecordingConnectorContext(); + engineContext.storageProperties = Collections.singletonList( + new StubHadoopStorageProperties(gcsConfig, "GCP")); + try (PaimonConnector connector = new PaimonConnector(properties, engineContext)) { + Method ensureCatalog = PaimonConnector.class.getDeclaredMethod("ensureCatalog"); + ensureCatalog.setAccessible(true); + Catalog catalog = (Catalog) ensureCatalog.invoke(connector); + RESTCatalog rest = PaimonCatalogOps.CatalogBackedPaimonCatalogOps.restCatalog(catalog); + Assertions.assertNotNull(rest); + CatalogContext context = rest.catalogLoader().context(); + gcsConfig.forEach((key, value) -> Assertions.assertEquals(value, context.hadoopConf().get(key))); + Assertions.assertInstanceOf(NativeGcsFileIO.Loader.class, context.preferIO()); + + org.apache.paimon.fs.Path path = new org.apache.paimon.fs.Path("s3a://bucket/private-table"); + Method dataFileIO = RESTCatalog.class.getDeclaredMethod("fileIOForData", + org.apache.paimon.fs.Path.class, Identifier.class); + dataFileIO.setAccessible(true); + Object fileIO = dataFileIO.invoke(rest, path, Identifier.create("db", "table")); + if (dataTokenEnabled) { + Assertions.assertInstanceOf(RESTTokenFileIO.class, fileIO); + } else { + Assertions.assertInstanceOf(ResolvingFileIO.class, fileIO); + } + // External table paths use this context even when REST data tokens are enabled. + Assertions.assertInstanceOf(NativeGcsFileIO.class, context.preferIO().load(path)); + } + } + } finally { + server.stop(0); + } + } + @Test public void filesystemSetsMetastoreFilesystemAndWarehouse() { Options opts = PaimonCatalogFactory.buildCatalogOptions( From 0f59c5dc97b283d5829c99cce3235f821cf0eea9 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 9 Oct 2026 23:48:47 +0800 Subject: [PATCH 21/47] [fix](connector) Defer native GCP support for Paimon and Hudi ### What problem does this PR solve? Related PR: #68540 Problem Summary: Catalog-wide native GCP adapters rewrote independent S3 locations in Paimon and Hudi to GCS. Defer these integrations, restore the original catalog/FileIO and scanner packaging paths, and remove the shared Paimon FileIO module. Reject resolved native GCP authentication before connector clients are created, including Paimon CREATE/ALTER and connector rebuilds. Keep GCS HMAC, explicit anonymous access, and existing S3 behavior. Shared GCP and Iceberg code and dependencies remain unchanged. ### Release note Native GCP authentication for Paimon and Hudi is deferred. Configurations selecting ADC, Compute Engine credentials, or service-account impersonation fail with an explicit unsupported-authentication error. ### Check List (For Author) - Test: Added authentication rejection/compatibility unit coverage and Paimon CREATE/ALTER validation coverage. Checkstyle passed for both FE connectors and both JNI scanners; connector import and metadata-funnel gates, changed-POM parsing, scope checks, and git diff --check passed. Unit tests and compilation were not run: this checkout lacks thirdparty/installed/bin/protoc and the available working validation JDK is 20, while the FE runner requires 17. - Behavior changed: Yes, native GCP is explicitly unsupported for Paimon/Hudi; existing HMAC/S3 behavior is preserved. - Does this need documentation: Yes, update the PR scope and deployment notes. --- .../hadoop-hudi-scanner/pom.xml | 7 - .../doris/hudi/HadoopHudiPluginTest.java | 11 -- fe/be-java-extensions/paimon-scanner/pom.xml | 13 -- .../apache/doris/paimon/PaimonPluginTest.java | 25 ---- .../connector/hudi/HudiCatalogProperties.java | 7 + .../connector/hudi/HudiConnectorMetadata.java | 15 +-- .../hudi/HudiCatalogPropertiesTest.java | 58 +++++++++ .../connector/hudi/HudiTableTypeTest.java | 45 ------- fe/fe-connector/fe-connector-paimon/pom.xml | 6 - .../paimon/PaimonCatalogFactory.java | 14 -- .../paimon/PaimonCatalogProperties.java | 7 + .../connector/paimon/PaimonConnector.java | 23 ++-- .../paimon/PaimonCatalogFactoryTest.java | 90 ------------- .../paimon/PaimonCatalogPropertiesTest.java | 58 +++++++++ ...PaimonConnectorValidatePropertiesTest.java | 17 +++ fe/fe-paimon-common/pom.xml | 50 -------- .../apache/doris/paimon/NativeGcsFileIO.java | 121 ------------------ .../doris/paimon/NativeGcsFileIOTest.java | 106 --------------- fe/pom.xml | 1 - 19 files changed, 159 insertions(+), 515 deletions(-) delete mode 100644 fe/fe-paimon-common/pom.xml delete mode 100644 fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java delete mode 100644 fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml b/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml index 835b2dff5afefa..525d763030bc23 100644 --- a/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml +++ b/fe/be-java-extensions/hadoop-hudi-scanner/pom.xml @@ -43,13 +43,6 @@ under the License. - - - com.google.cloud.bigdataoss - gcs-connector - shaded - runtime - org.apache.doris jni-spi diff --git a/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java b/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java index a099fc46626c53..bc40bef8eee19f 100644 --- a/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java +++ b/fe/be-java-extensions/hadoop-hudi-scanner/src/test/java/org/apache/doris/hudi/HadoopHudiPluginTest.java @@ -111,16 +111,6 @@ private static DorisPlugin loadPlugin() { } /** The path the plugin registry takes: services file, plugin class, factory list. */ - @Test - public void instantiatesNativeGcsFilesystem() throws Exception { - Configuration conf = new Configuration(); - conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); - try (FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance()) { - Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", - fs.getClass().getName()); - } - } - @Test public void isDiscoverableThroughServiceLoader() { Assertions.assertTrue(loadPlugin() instanceof HadoopHudiPlugin); @@ -174,7 +164,6 @@ public void resolvesEveryFilesystemSchemeAScanCanArriveOn() throws IOException { expected.put("hdfs", "org.apache.hadoop.hdfs.DistributedFileSystem"); expected.put("webhdfs", "org.apache.hadoop.hdfs.web.WebHdfsFileSystem"); expected.put("s3a", "org.apache.hadoop.fs.s3a.S3AFileSystem"); - expected.put("gs", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); expected.put("obs", "org.apache.hadoop.fs.obs.OBSFileSystem"); expected.put("abfs", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem"); expected.put("wasb", "org.apache.hadoop.fs.azure.NativeAzureFileSystem"); diff --git a/fe/be-java-extensions/paimon-scanner/pom.xml b/fe/be-java-extensions/paimon-scanner/pom.xml index 8f29fb2c5f73cd..b69e15eb3c0b2e 100644 --- a/fe/be-java-extensions/paimon-scanner/pom.xml +++ b/fe/be-java-extensions/paimon-scanner/pom.xml @@ -41,19 +41,6 @@ under the License. - - - org.apache.doris - fe-paimon-common - ${revision} - - - - com.google.cloud.bigdataoss - gcs-connector - shaded - runtime - org.apache.doris jni-spi diff --git a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java index 7434aceeb76e31..0d2b8c768d57c8 100644 --- a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java +++ b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonPluginTest.java @@ -94,30 +94,6 @@ private static DorisPlugin loadPlugin() { } /** The path the plugin registry takes: services file, plugin class, factory list. */ - @Test - public void reconstructsNativeGcsFileIO() throws Exception { - NativeGcsFileIO fileIO = new NativeGcsFileIO(new org.apache.paimon.fs.Path("s3a://bucket/warehouse")); - Configuration conf = new Configuration(false); - conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); - conf.set("fs.gs.auth.type", "APPLICATION_DEFAULT"); - fileIO.configure(CatalogContext.create(new Options(), conf)); - NativeGcsFileIO restored = InstantiationUtil.deserializeObject( - InstantiationUtil.serializeObject(fileIO), getClass().getClassLoader()); - Assertions.assertEquals("APPLICATION_DEFAULT", restored.hadoopConf().get("fs.gs.auth.type")); - Assertions.assertEquals("gs://bucket/file", NativeGcsFileIO.normalize( - new org.apache.paimon.fs.Path("s3a://bucket/file")).toString()); - } - - @Test - public void instantiatesNativeGcsFilesystem() throws Exception { - Configuration conf = new Configuration(); - conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); - try (FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance()) { - Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", - fs.getClass().getName()); - } - } - @Test public void isDiscoverableThroughServiceLoader() { Assertions.assertTrue(loadPlugin() instanceof PaimonPlugin); @@ -181,7 +157,6 @@ public void resolvesEveryFilesystemSchemeAScanCanArriveOn() throws IOException { expected.put("hdfs", "org.apache.hadoop.hdfs.DistributedFileSystem"); expected.put("webhdfs", "org.apache.hadoop.hdfs.web.WebHdfsFileSystem"); expected.put("s3a", "org.apache.hadoop.fs.s3a.S3AFileSystem"); - expected.put("gs", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); expected.put("obs", "org.apache.hadoop.fs.obs.OBSFileSystem"); expected.put("abfs", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem"); expected.put("wasb", "org.apache.hadoop.fs.azure.NativeAzureFileSystem"); diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java index 8eea98b52331ff..6336e16850e85e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hudi; import org.apache.doris.connector.hms.HmsClientConfig; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import org.apache.doris.foundation.property.ConnectorProperty; import org.apache.doris.foundation.property.ParamRules; @@ -97,6 +98,12 @@ private HudiCatalogProperties(Map properties) { } public static HudiCatalogProperties of(Map properties) { + // Reject unsupported native auth before creating clients, including catalog rebuilds. + // Resolve the mode centrally so GCS HMAC and existing S3 configurations remain supported. + if (GcsAuthResolver.resolve(properties).flatMap(auth -> auth.getNativeCredential()).isPresent()) { + throw new IllegalArgumentException("Native GCP authentication is not supported for Hudi yet; " + + "use GCS HMAC credentials instead."); + } HudiCatalogProperties p = new HudiCatalogProperties(properties); ConnectorPropertiesUtils.bindConnectorProperties(p, properties); new ParamRules() diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 7863b04b856e3f..0599c6efb05eb6 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -203,7 +203,7 @@ public Optional getTableHandle( return Optional.empty(); } HmsTableInfo tableInfo = hmsClient.getTable(dbName, tableName); - String location = hadoopTableLocation(tableInfo.getLocation()); + String location = tableInfo.getLocation(); String hudiTableType = detectHudiTableType(tableInfo); // Extract partition key names @@ -222,19 +222,6 @@ public Optional getTableHandle( .build()); } - private String hadoopTableLocation(String location) { - // Native GCP auth configures fs.gs.*, while HMS may retain an S3-compatible location. - // Normalize the handle so metadata, split planning and the JNI reader share the same base path. - if ("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem".equals(storageHadoopConfig.get("fs.gs.impl"))) { - if (location.regionMatches(true, 0, "s3://", 0, 5)) { - return "gs://" + location.substring(5); - } else if (location.regionMatches(true, 0, "s3a://", 0, 6)) { - return "gs://" + location.substring(6); - } - } - return location; - } - @Override public Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java index ba2630f737d0ea..7f14bc6aae37fc 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java @@ -137,4 +137,62 @@ void rawIsAnImmutableCopyOfTheInput() { Assertions.assertFalse(p.getRaw().containsKey("added_afterwards")); Assertions.assertThrows(UnsupportedOperationException.class, () -> p.getRaw().put("k", "v")); } + + @Test + public void rejectsNativeGcpAuthenticationBeforeClientCreation() { + for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + for (boolean impersonate : new boolean[] {false, true}) { + Map properties = HudiTestProperties.minimalMap(); + properties.put("provider", "GCP"); + properties.put("gs.credential_provider_type", mode); + if (impersonate) { + properties.put("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"); + } + IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, + () -> HudiCatalogProperties.of(properties)); + Assertions.assertTrue(error.getMessage().contains( + "Native GCP authentication is not supported for Hudi")); + } + } + } + + @Test + public void rejectsImplicitGcpAdcAndImpersonationOnly() { + for (Map selection : java.util.List.of( + Map.of("provider", "GCP"), + Map.of("warehouse", "gs://bucket/warehouse"), + Map.of("gs.endpoint", "https://storage.googleapis.com"), + Map.of("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"))) { + Map properties = HudiTestProperties.minimalMap(); + properties.putAll(selection); + Assertions.assertThrows(IllegalArgumentException.class, + () -> HudiCatalogProperties.of(properties)); + } + } + + @Test + public void preservesGcsHmacAndS3Authentication() { + for (String provider : new String[] {"GCP", "S3"}) { + for (String prefix : new String[] {"gs.", "s3."}) { + Map properties = HudiTestProperties.minimalMap(); + properties.put("provider", provider); + properties.put("s3.endpoint", "https://storage.googleapis.com"); + properties.put(prefix + "access_key", "test-access-key"); + properties.put(prefix + "secret_key", "test-secret-key"); + Assertions.assertDoesNotThrow(() -> HudiCatalogProperties.of(properties)); + } + } + Map s3 = HudiTestProperties.minimalMap(); + s3.put("provider", "S3"); + s3.put("s3.role_arn", "arn:aws:iam::123456789012:role/test-role"); + Assertions.assertDoesNotThrow(() -> HudiCatalogProperties.of(s3)); + } + + @Test + public void preservesExplicitAnonymousGcsAccess() { + Map properties = HudiTestProperties.minimalMap(); + properties.put("provider", "GCP"); + properties.put("gs.credential_provider_type", "ANONYMOUS"); + Assertions.assertDoesNotThrow(() -> HudiCatalogProperties.of(properties)); + } } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java index af33f7e10d7e65..718717b7c9f2e9 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java @@ -85,51 +85,6 @@ public void testUnknownWhenNoHudiSignal() { detect("org.apache.hadoop.mapred.TextInputFormat", Collections.emptyMap())); } - @Test - public void testNativeGcpLocationsSelectGcsForCowAndMor() { - for (String authType : List.of("APPLICATION_DEFAULT", "COMPUTE_ENGINE")) { - for (String inputFormat : List.of("org.apache.hudi.hadoop.HoodieParquetInputFormat", - "org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat")) { - Map config = Map.of( - "fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", - "fs.gs.auth.type", authType, - "fs.gs.auth.impersonation.service.account", "target@project.iam.gserviceaccount.com"); - for (String scheme : List.of("s3", "s3a", "S3", "S3A", "gs")) { - HudiTableHandle handle = tableHandle(scheme + "://bucket/table%20name/", inputFormat, config); - Assertions.assertEquals("gs://bucket/table%20name/", handle.getBasePath()); - } - for (String location : List.of("hdfs://namenode/table", "file:///tmp/table")) { - Assertions.assertEquals(location, tableHandle(location, inputFormat, config).getBasePath()); - } - } - } - } - - @Test - public void testLegacyLocationsKeepTheirScheme() { - // GCS HMAC and anonymous access use S3A; ordinary AWS/HDFS configs must also stay unchanged. - for (Map config : List.of(Collections.emptyMap(), - Map.of("fs.gs.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"), - Map.of("fs.s3a.access.key", "access-key", "fs.s3a.secret.key", "secret-key"))) { - for (String location : List.of("s3://bucket/table", "s3a://bucket/table", "gs://bucket/table")) { - Assertions.assertEquals(location, tableHandle(location, - "org.apache.hudi.hadoop.HoodieParquetInputFormat", config).getBasePath()); - } - } - } - - private HudiTableHandle tableHandle(String location, String inputFormat, Map config) { - HmsTableInfo info = HmsTableInfo.builder() - .dbName("db").tableName("t") - .location(location) - .inputFormat(inputFormat) - .parameters(Collections.emptyMap()) - .build(); - HudiConnectorMetadata metadata = new HudiConnectorMetadata(new FakeHmsClient(info), - HudiTestProperties.minimal(), new DirectHudiMetaClientExecutor(), config); - return (HudiTableHandle) metadata.getTableHandle(null, "db", "t").orElseThrow(); - } - /** * Minimal {@link HmsClient} double returning a fixed table. Only {@code tableExists} * and {@code getTable} are exercised by {@code getTableHandle}; the rest fail loud. diff --git a/fe/fe-connector/fe-connector-paimon/pom.xml b/fe/fe-connector/fe-connector-paimon/pom.xml index 1686a3c38de3ec..9c332e6c19b465 100644 --- a/fe/fe-connector/fe-connector-paimon/pom.xml +++ b/fe/fe-connector/fe-connector-paimon/pom.xml @@ -41,12 +41,6 @@ under the License. - - - org.apache.doris - fe-paimon-common - ${revision} - ${project.groupId} fe-connector-spi diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java index a8fe652967669b..e6c96cdab9d4ec 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogFactory.java @@ -23,13 +23,11 @@ import org.apache.doris.connector.metastore.paimon.rest.PaimonRestMetaStoreProperties; import org.apache.doris.connector.metastore.spi.AbstractMetaStoreProperties; import org.apache.doris.connector.metastore.spi.MetaStoreProviders; -import org.apache.doris.paimon.NativeGcsFileIO; import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.paimon.catalog.CatalogContext; import org.apache.paimon.catalog.FileSystemCatalogFactory; import org.apache.paimon.jdbc.JdbcCatalogFactory; import org.apache.paimon.options.CatalogOptions; @@ -113,18 +111,6 @@ public static Options buildCatalogOptions(PaimonCatalogProperties catalogPropert return options; } - static CatalogContext createCatalogContext(Options options, Configuration conf) { - if (!"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem".equals(conf.get("fs.gs.impl"))) { - return CatalogContext.create(options, conf); - } - String warehouse = options.get(CatalogOptions.WAREHOUSE); - if (warehouse != null) { - options.set(CatalogOptions.WAREHOUSE, - NativeGcsFileIO.normalize(new org.apache.paimon.fs.Path(warehouse)).toString()); - } - return CatalogContext.create(options, conf, new NativeGcsFileIO.Loader(), null); - } - static boolean isCatalogCacheEnabled(PaimonCatalogProperties catalogProperties) { return assembleCatalogOptions(catalogProperties).get(CatalogOptions.CACHE_ENABLED); } diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java index 18215a3858021b..9df7f85f732f15 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java @@ -19,6 +19,7 @@ import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.metastore.spi.MetaStoreProviders; +import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import org.apache.doris.foundation.property.ConnectorProperty; @@ -120,6 +121,12 @@ private PaimonCatalogProperties(Map properties) { * overwrite a key, never remove one, so a rejected unknown key could not be repaired. */ public static PaimonCatalogProperties of(Map properties) { + // Reject unsupported native auth before creating clients, including catalog rebuilds. + // Resolve the mode centrally so GCS HMAC and existing S3 configurations remain supported. + if (GcsAuthResolver.resolve(properties).flatMap(auth -> auth.getNativeCredential()).isPresent()) { + throw new IllegalArgumentException("Native GCP authentication is not supported for Paimon yet; " + + "use GCS HMAC credentials instead."); + } PaimonCatalogProperties p = new PaimonCatalogProperties(properties); ConnectorPropertiesUtils.bindConnectorProperties(p, properties); p.flavor = p.catalogType.toLowerCase(Locale.ROOT); diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java index 1646f2b650f024..967196a550a00a 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java @@ -428,8 +428,9 @@ private Catalog createCatalog() { // Canonical storage config from the FE-bound fe-filesystem StorageProperties (P1-T03), replacing // the legacy buildObjectStorageHadoopConfig path: object stores contribute their fs.s3a.*/fs.oss.* // /fs.cosn.*/fs.obs.* translation, and an HDFS-backed catalog contributes its hadoop.config.resources - // XML + HA + auth keys (C2; the defaults-free fe-filesystem Hadoop map). Empty for a catalog - // with no typed storage at all (it reaches the conf via the raw fs./dfs./hadoop. passthrough). + // XML + HA + auth keys (C2; the defaults-free fe-filesystem Hadoop map). Empty for REST (the server + // owns storage) and for a catalog with no typed storage at all (it reaches the conf via the raw + // fs./dfs./hadoop. passthrough). Map storageHadoopConfig = buildStorageHadoopConfig(); switch (flavor) { @@ -437,21 +438,19 @@ private Catalog createCatalog() { // filesystem carries a Hadoop Configuration for HDFS/S3 storage. Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration( catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, conf), flavor, + return createCatalogFromContext(CatalogContext.create(options, conf), flavor, "Failed to create Paimon catalog with filesystem metastore"); } case PaimonCatalogProperties.REST: { - // External table paths and reads without REST data tokens use local FileIO. - Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration( - catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, conf), flavor, + // rest is Options-only (no storage Configuration; the REST server owns storage). + return createCatalogFromContext(CatalogContext.create(options), flavor, "Failed to create Paimon catalog with REST metastore"); } case PaimonCatalogProperties.JDBC: { maybeRegisterJdbcDriver(); Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration( catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, conf), flavor, + return createCatalogFromContext(CatalogContext.create(options, conf), flavor, "Failed to create Paimon catalog with JDBC metastore"); } case PaimonCatalogProperties.HMS: { @@ -481,7 +480,7 @@ private Catalog createCatalog() { options.set("client-pool-cache.keys", appendHmsCacheKeys( options.get("client-pool-cache.keys"))); HadoopAuthenticator hmsAuth = buildHmsAuthenticator(catalogProps.getRaw(), storageHadoopConfig); - return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, hc), flavor, + return createCatalogFromContext(CatalogContext.create(options, hc), flavor, hmsAuth, storageHadoopConfig, "Failed to create Paimon catalog with HMS metastore"); } @@ -498,7 +497,7 @@ private Catalog createCatalog() { dlfConf.put(PaimonCatalogFactory.DLF_CLIENT_POOL_IDENTITY, PaimonCatalogFactory.dlfClientPoolIdentity(dlfConf)); HiveConf hc = PaimonCatalogFactory.assembleHiveConf(null, dlfConf); - return createCatalogFromContext(PaimonCatalogFactory.createCatalogContext(options, hc), flavor, + return createCatalogFromContext(CatalogContext.create(options, hc), flavor, "Failed to create Paimon catalog with DLF metastore"); } default: @@ -525,8 +524,8 @@ Options buildCatalogOptions() { * defaults-free so it never clobbers a co-bound object-store provider's tuned fs.s3a.* here). This * replaces the legacy {@code StorageProperties.buildObjectStorageHadoopConfig(properties)} call that * {@link PaimonCatalogFactory#buildHadoopConfiguration}/{@code buildHmsHiveConf} - * used to make. Empty for a catalog with no typed storage (it reaches the conf via the raw - * fs./dfs./hadoop. passthrough). + * used to make. Empty for REST (the server owns storage) and for a catalog with no typed storage (it + * reaches the conf via the raw fs./dfs./hadoop. passthrough). */ // Package-private (not private) so PaimonCatalogFactoryTest can drive the storage().getStorageProperties() // -> toHadoopProperties() -> Configuration wiring end-to-end (visible for testing). diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java index 22776b364190f7..525f4855b9fa0f 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogFactoryTest.java @@ -21,27 +21,17 @@ import org.apache.doris.filesystem.properties.HadoopStorageProperties; import org.apache.doris.filesystem.properties.StorageKind; import org.apache.doris.filesystem.properties.StorageProperties; -import org.apache.doris.paimon.NativeGcsFileIO; -import com.sun.net.httpserver.HttpServer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.paimon.catalog.Catalog; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.catalog.Identifier; -import org.apache.paimon.fs.ResolvingFileIO; import org.apache.paimon.options.CatalogOptions; import org.apache.paimon.options.Options; -import org.apache.paimon.rest.RESTCatalog; -import org.apache.paimon.rest.RESTTokenFileIO; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import java.lang.reflect.Method; -import java.net.InetSocketAddress; import java.net.URL; import java.net.URLClassLoader; -import java.nio.charset.StandardCharsets; import java.util.Collections; import java.util.HashMap; import java.util.Map; @@ -90,86 +80,6 @@ private static Map storage(String... kv) { // buildCatalogOptions — per-flavor metastore identifier + warehouse // --------------------------------------------------------------------- - @Test - public void nativeGcsNormalizesWarehouseAndInstallsAliasFileIO() { - for (String scheme : new String[] {"s3", "s3a", "gs"}) { - Options options = new Options(); - options.set(CatalogOptions.WAREHOUSE, scheme + "://bucket/private-warehouse"); - Configuration conf = new Configuration(false); - conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); - conf.set("fs.gs.auth.type", "APPLICATION_DEFAULT"); - CatalogContext context = PaimonCatalogFactory.createCatalogContext(options, conf); - Assertions.assertEquals("gs://bucket/private-warehouse", context.options().get(CatalogOptions.WAREHOUSE)); - Assertions.assertInstanceOf(NativeGcsFileIO.class, - context.preferIO().load(new org.apache.paimon.fs.Path(scheme + "://bucket/persisted-file"))); - Assertions.assertEquals("APPLICATION_DEFAULT", context.hadoopConf().get("fs.gs.auth.type")); - } - Options legacy = new Options(); - legacy.set(CatalogOptions.WAREHOUSE, "s3a://bucket/warehouse"); - Configuration conf = new Configuration(false); - conf.set("fs.gs.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"); - CatalogContext context = PaimonCatalogFactory.createCatalogContext(legacy, conf); - Assertions.assertNull(context.preferIO()); - Assertions.assertEquals("s3a://bucket/warehouse", context.options().get(CatalogOptions.WAREHOUSE)); - } - - @Test - public void restCatalogRetainsNativeGcsContextAndTokenSelection() throws Exception { - HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); - server.createContext("/v1/config", exchange -> { - byte[] response = "{\"defaults\":{},\"overrides\":{}}".getBytes(StandardCharsets.UTF_8); - exchange.getResponseHeaders().set("Content-Type", "application/json"); - exchange.sendResponseHeaders(200, response.length); - try (java.io.OutputStream output = exchange.getResponseBody()) { - output.write(response); - } - }); - server.start(); - try { - for (boolean dataTokenEnabled : new boolean[] {false, true}) { - Map properties = props( - "paimon.catalog.type", "rest", - "paimon.rest.uri", "http://127.0.0.1:" + server.getAddress().getPort(), - "paimon.rest.prefix", "test-prefix", - "paimon.rest.token.provider", "bear", - "paimon.rest.token", "test-token", - "paimon.rest.data-token.enabled", Boolean.toString(dataTokenEnabled)); - Map gcsConfig = storage( - "fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", - "fs.gs.auth.type", "APPLICATION_DEFAULT", - "fs.gs.auth.impersonation.service.account", "reader@project.iam.gserviceaccount.com"); - RecordingConnectorContext engineContext = new RecordingConnectorContext(); - engineContext.storageProperties = Collections.singletonList( - new StubHadoopStorageProperties(gcsConfig, "GCP")); - try (PaimonConnector connector = new PaimonConnector(properties, engineContext)) { - Method ensureCatalog = PaimonConnector.class.getDeclaredMethod("ensureCatalog"); - ensureCatalog.setAccessible(true); - Catalog catalog = (Catalog) ensureCatalog.invoke(connector); - RESTCatalog rest = PaimonCatalogOps.CatalogBackedPaimonCatalogOps.restCatalog(catalog); - Assertions.assertNotNull(rest); - CatalogContext context = rest.catalogLoader().context(); - gcsConfig.forEach((key, value) -> Assertions.assertEquals(value, context.hadoopConf().get(key))); - Assertions.assertInstanceOf(NativeGcsFileIO.Loader.class, context.preferIO()); - - org.apache.paimon.fs.Path path = new org.apache.paimon.fs.Path("s3a://bucket/private-table"); - Method dataFileIO = RESTCatalog.class.getDeclaredMethod("fileIOForData", - org.apache.paimon.fs.Path.class, Identifier.class); - dataFileIO.setAccessible(true); - Object fileIO = dataFileIO.invoke(rest, path, Identifier.create("db", "table")); - if (dataTokenEnabled) { - Assertions.assertInstanceOf(RESTTokenFileIO.class, fileIO); - } else { - Assertions.assertInstanceOf(ResolvingFileIO.class, fileIO); - } - // External table paths use this context even when REST data tokens are enabled. - Assertions.assertInstanceOf(NativeGcsFileIO.class, context.preferIO().load(path)); - } - } - } finally { - server.stop(0); - } - } - @Test public void filesystemSetsMetastoreFilesystemAndWarehouse() { Options opts = PaimonCatalogFactory.buildCatalogOptions( diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java index 1e97a092030cfd..a41d2520024cfa 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java @@ -173,4 +173,62 @@ public void rawMapIsNotWritable() { Assertions.assertThrows(UnsupportedOperationException.class, () -> p.getRaw().put("k", "v")); } + + @Test + public void rejectsNativeGcpAuthenticationBeforeClientCreation() { + for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + for (boolean impersonate : new boolean[] {false, true}) { + Map properties = props("warehouse", "s3a://bucket/warehouse"); + properties.put("provider", "GCP"); + properties.put("gs.credential_provider_type", mode); + if (impersonate) { + properties.put("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"); + } + IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, + () -> PaimonCatalogProperties.of(properties)); + Assertions.assertTrue(error.getMessage().contains( + "Native GCP authentication is not supported for Paimon")); + } + } + } + + @Test + public void rejectsImplicitGcpAdcAndImpersonationOnly() { + for (Map selection : java.util.List.of( + Map.of("provider", "GCP"), + Map.of("warehouse", "gs://bucket/warehouse"), + Map.of("gs.endpoint", "https://storage.googleapis.com"), + Map.of("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"))) { + Map properties = props("warehouse", "s3a://bucket/warehouse"); + properties.putAll(selection); + Assertions.assertThrows(IllegalArgumentException.class, + () -> PaimonCatalogProperties.of(properties)); + } + } + + @Test + public void preservesGcsHmacAndS3Authentication() { + for (String provider : new String[] {"GCP", "S3"}) { + for (String prefix : new String[] {"gs.", "s3."}) { + Map properties = props("warehouse", "s3a://bucket/warehouse"); + properties.put("provider", provider); + properties.put("s3.endpoint", "https://storage.googleapis.com"); + properties.put(prefix + "access_key", "test-access-key"); + properties.put(prefix + "secret_key", "test-secret-key"); + Assertions.assertDoesNotThrow(() -> PaimonCatalogProperties.of(properties)); + } + } + Map s3 = props("warehouse", "s3a://bucket/warehouse"); + s3.put("provider", "S3"); + s3.put("s3.role_arn", "arn:aws:iam::123456789012:role/test-role"); + Assertions.assertDoesNotThrow(() -> PaimonCatalogProperties.of(s3)); + } + + @Test + public void preservesExplicitAnonymousGcsAccess() { + Map properties = props("warehouse", "s3a://bucket/warehouse"); + properties.put("provider", "GCP"); + properties.put("gs.credential_provider_type", "ANONYMOUS"); + Assertions.assertDoesNotThrow(() -> PaimonCatalogProperties.of(properties)); + } } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java index 991cdffd535aae..a4a3287362d6ab 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java @@ -215,4 +215,21 @@ public void dlfCatalogTypeValidatesLegacyAliasOnlyProperties() { "dlf.secret_key", "sk", "dlf.region", "cn-hangzhou")); } + + @Test + public void rejectsNativeGcpOnCreateAndAlter() { + Map nativeGcp = props("provider", "GCP", + "gs.credential_provider_type", "COMPUTE_ENGINE", + "gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"); + Map properties = props("warehouse", "s3a://bucket/warehouse"); + properties.putAll(nativeGcp); + IllegalArgumentException createError = Assertions.assertThrows(IllegalArgumentException.class, + () -> PROVIDER.validateProperties(properties)); + Assertions.assertTrue(createError.getMessage().contains( + "Native GCP authentication is not supported for Paimon")); + IllegalArgumentException alterError = Assertions.assertThrows(IllegalArgumentException.class, + () -> PROVIDER.validatePropertiesForUpdate(props("warehouse", "s3a://bucket/warehouse"), nativeGcp)); + Assertions.assertTrue(alterError.getMessage().contains( + "Native GCP authentication is not supported for Paimon")); + } } diff --git a/fe/fe-paimon-common/pom.xml b/fe/fe-paimon-common/pom.xml deleted file mode 100644 index 52dd2132e06390..00000000000000 --- a/fe/fe-paimon-common/pom.xml +++ /dev/null @@ -1,50 +0,0 @@ - - - - 4.0.0 - - org.apache.doris - fe - ${revision} - ../pom.xml - - fe-paimon-common - Doris Shared Paimon FileIO - GCS URI normalization shared by FE planning and serialized BE Paimon readers. - - - - org.apache.paimon - paimon-common - provided - - - org.apache.hadoop - hadoop-common - provided - - - org.junit.jupiter - junit-jupiter - test - - - diff --git a/fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java b/fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java deleted file mode 100644 index b0644d73128f3c..00000000000000 --- a/fe/fe-paimon-common/src/main/java/org/apache/doris/paimon/NativeGcsFileIO.java +++ /dev/null @@ -1,121 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.paimon; - -import org.apache.paimon.fs.FileIO; -import org.apache.paimon.fs.FileIOLoader; -import org.apache.paimon.fs.FileStatus; -import org.apache.paimon.fs.Path; -import org.apache.paimon.fs.PositionOutputStream; -import org.apache.paimon.fs.RemoteIterator; -import org.apache.paimon.fs.SeekableInputStream; -import org.apache.paimon.fs.hadoop.HadoopFileIO; - -import java.io.IOException; - -/** Routes legacy GCS aliases through the native Hadoop GCS connector, including persisted file paths. */ -public class NativeGcsFileIO extends HadoopFileIO { - private static final long serialVersionUID = 1L; - - public NativeGcsFileIO(Path path) { - super(normalize(path)); - } - - public static Path normalize(Path path) { - String scheme = path.toUri().getScheme(); - if ("s3".equalsIgnoreCase(scheme) || "s3a".equalsIgnoreCase(scheme)) { - return new Path("gs" + path.toString().substring(scheme.length())); - } - return path; - } - - @Override - public boolean isObjectStore() { - return true; - } - - @Override - public SeekableInputStream newInputStream(Path path) throws IOException { - return super.newInputStream(normalize(path)); - } - - @Override - public PositionOutputStream newOutputStream(Path path, boolean overwrite) throws IOException { - return super.newOutputStream(normalize(path), overwrite); - } - - @Override - public FileStatus getFileStatus(Path path) throws IOException { - return super.getFileStatus(normalize(path)); - } - - @Override - public FileStatus[] listStatus(Path path) throws IOException { - return super.listStatus(normalize(path)); - } - - @Override - public RemoteIterator listFilesIterative(Path path, boolean recursive) throws IOException { - return super.listFilesIterative(normalize(path), recursive); - } - - @Override - public boolean exists(Path path) throws IOException { - return super.exists(normalize(path)); - } - - @Override - public boolean delete(Path path, boolean recursive) throws IOException { - return super.delete(normalize(path), recursive); - } - - @Override - public boolean mkdirs(Path path) throws IOException { - return super.mkdirs(normalize(path)); - } - - @Override - public boolean rename(Path source, Path target) throws IOException { - return super.rename(normalize(source), normalize(target)); - } - - @Override - public void overwriteFileUtf8(Path path, String content) throws IOException { - super.overwriteFileUtf8(normalize(path), content); - } - - @Override - public boolean tryAtomicOverwriteViaRename(Path path, String content) throws IOException { - return super.tryAtomicOverwriteViaRename(normalize(path), content); - } - - public static final class Loader implements FileIOLoader { - private static final long serialVersionUID = 1L; - - @Override - public String getScheme() { - return "gs"; - } - - @Override - public FileIO load(Path path) { - return "gs".equalsIgnoreCase(normalize(path).toUri().getScheme()) - ? new NativeGcsFileIO(path) : new HadoopFileIO(path); - } - } -} diff --git a/fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java b/fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java deleted file mode 100644 index f9d235f403df41..00000000000000 --- a/fe/fe-paimon-common/src/test/java/org/apache/doris/paimon/NativeGcsFileIOTest.java +++ /dev/null @@ -1,106 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.paimon; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.BlockLocation; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.RawLocalFileSystem; -import org.apache.paimon.catalog.CatalogContext; -import org.apache.paimon.fs.Path; -import org.apache.paimon.options.Options; -import org.apache.paimon.utils.InstantiationUtil; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; - -import java.io.File; -import java.io.IOException; -import java.net.URI; - -public class NativeGcsFileIOTest { - @TempDir - public java.nio.file.Path directory; - - @Test - public void aliasesUseNativeFilesystemBeforeAndAfterSerialization() throws Exception { - Configuration conf = new Configuration(false); - conf.set("fs.gs.impl", LocalGcsFileSystem.class.getName()); - conf.setBoolean("fs.gs.impl.disable.cache", true); - conf.set("test.gcs.root", directory.toString()); - conf.set("fs.gs.auth.type", "APPLICATION_DEFAULT"); - NativeGcsFileIO original = new NativeGcsFileIO(new Path("s3a://bucket/warehouse")); - original.configure(CatalogContext.create(new Options(), conf)); - NativeGcsFileIO restored = InstantiationUtil.deserializeObject( - InstantiationUtil.serializeObject(original), getClass().getClassLoader()); - for (NativeGcsFileIO fileIO : new NativeGcsFileIO[] {original, restored}) { - Assertions.assertEquals("APPLICATION_DEFAULT", fileIO.hadoopConf().get("fs.gs.auth.type")); - Assertions.assertTrue(fileIO.isObjectStore()); - for (String scheme : new String[] {"s3", "s3a", "gs"}) { - Path folder = new Path(scheme + "://bucket/" + scheme); - Path file = new Path(folder, "manifest"); - Assertions.assertTrue(fileIO.mkdirs(folder)); - fileIO.writeFile(file, "private-gcs-data", true); - Assertions.assertEquals("private-gcs-data", fileIO.readFileUtf8(file)); - Assertions.assertTrue(fileIO.exists(file)); - Assertions.assertEquals(16, fileIO.getFileStatus(file).getLen()); - Assertions.assertEquals(1, fileIO.listStatus(folder).length); - Assertions.assertTrue(fileIO.listFilesIterative(folder, true).hasNext()); - Path target = new Path("s3a://bucket/" + scheme + "/renamed"); - Assertions.assertTrue(fileIO.rename(file, target)); - Assertions.assertTrue(fileIO.delete(target, false)); - } - } - } - - @Test - public void normalizationPreservesObjectKeyAndOtherSchemes() { - Assertions.assertEquals("gs://bucket/a%20b/part-1", - NativeGcsFileIO.normalize(new Path("s3a://bucket/a%20b/part-1")).toString()); - for (String uri : new String[] {"gs://bucket/key", "hdfs://namenode/key", "file:/tmp/key"}) { - Path path = new Path(uri); - Assertions.assertSame(path, NativeGcsFileIO.normalize(path)); - } - } - - /** Local backing storage that rejects any path not normalized to gs before Hadoop dispatch. */ - public static class LocalGcsFileSystem extends RawLocalFileSystem { - @Override - public URI getUri() { - return URI.create("gs://bucket"); - } - - @Override - public FileStatus getFileStatus(org.apache.hadoop.fs.Path path) throws IOException { - FileStatus status = super.getFileStatus(path); - return new FileStatus(status.getLen(), status.isDirectory(), 1, 4096, - status.getModificationTime(), path); - } - - @Override - public BlockLocation[] getFileBlockLocations(FileStatus status, long start, long length) { - return new BlockLocation[0]; - } - - @Override - public File pathToFile(org.apache.hadoop.fs.Path path) { - Assertions.assertEquals("gs", path.toUri().getScheme()); - return new File(getConf().get("test.gcs.root"), path.toUri().getPath()); - } - } -} diff --git a/fe/pom.xml b/fe/pom.xml index 27b31a332dde9c..e5a4185e330445 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -231,7 +231,6 @@ under the License. fe-common fe-trino-connector-common fe-iceberg-common - fe-paimon-common fe-catalog fe-filesystem fe-extension-spi From 8c18a565ccc8dd5c850d0787252726fa66f53ead Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Sat, 10 Oct 2026 00:02:28 +0800 Subject: [PATCH 22/47] [fix](connector) Remove deferred Paimon and Hudi auth guards ### What problem does this PR solve? Related PR: #68540 Problem Summary: Keep Paimon and Hudi unchanged while their native GCP integration is deferred. Remove the newly added native-auth rejection checks and their tests, restoring these files to the PR base. The earlier removal of native FileIO, URI rewriting, and dedicated dependencies remains. ### Release note Paimon and Hudi retain their existing configuration handling; native GCP integration remains deferred to a follow-up change. ### Check List (For Author) - Test: Both connector Checkstyle checks and git diff --check passed. Verified all Paimon/Hudi paths match the PR base. No runtime tests run; this change only removes the new guards and their associated tests. - Behavior changed: Yes, remove newly introduced native-auth rejection. - Does this need documentation: Yes, align PR scope notes. --- .../connector/hudi/HudiCatalogProperties.java | 7 --- .../hudi/HudiCatalogPropertiesTest.java | 58 ------------------- .../paimon/PaimonCatalogProperties.java | 7 --- .../paimon/PaimonCatalogPropertiesTest.java | 58 ------------------- ...PaimonConnectorValidatePropertiesTest.java | 17 ------ 5 files changed, 147 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java index 6336e16850e85e..8eea98b52331ff 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiCatalogProperties.java @@ -18,7 +18,6 @@ package org.apache.doris.connector.hudi; import org.apache.doris.connector.hms.HmsClientConfig; -import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import org.apache.doris.foundation.property.ConnectorProperty; import org.apache.doris.foundation.property.ParamRules; @@ -98,12 +97,6 @@ private HudiCatalogProperties(Map properties) { } public static HudiCatalogProperties of(Map properties) { - // Reject unsupported native auth before creating clients, including catalog rebuilds. - // Resolve the mode centrally so GCS HMAC and existing S3 configurations remain supported. - if (GcsAuthResolver.resolve(properties).flatMap(auth -> auth.getNativeCredential()).isPresent()) { - throw new IllegalArgumentException("Native GCP authentication is not supported for Hudi yet; " - + "use GCS HMAC credentials instead."); - } HudiCatalogProperties p = new HudiCatalogProperties(properties); ConnectorPropertiesUtils.bindConnectorProperties(p, properties); new ParamRules() diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java index 7f14bc6aae37fc..ba2630f737d0ea 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiCatalogPropertiesTest.java @@ -137,62 +137,4 @@ void rawIsAnImmutableCopyOfTheInput() { Assertions.assertFalse(p.getRaw().containsKey("added_afterwards")); Assertions.assertThrows(UnsupportedOperationException.class, () -> p.getRaw().put("k", "v")); } - - @Test - public void rejectsNativeGcpAuthenticationBeforeClientCreation() { - for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { - for (boolean impersonate : new boolean[] {false, true}) { - Map properties = HudiTestProperties.minimalMap(); - properties.put("provider", "GCP"); - properties.put("gs.credential_provider_type", mode); - if (impersonate) { - properties.put("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"); - } - IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, - () -> HudiCatalogProperties.of(properties)); - Assertions.assertTrue(error.getMessage().contains( - "Native GCP authentication is not supported for Hudi")); - } - } - } - - @Test - public void rejectsImplicitGcpAdcAndImpersonationOnly() { - for (Map selection : java.util.List.of( - Map.of("provider", "GCP"), - Map.of("warehouse", "gs://bucket/warehouse"), - Map.of("gs.endpoint", "https://storage.googleapis.com"), - Map.of("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"))) { - Map properties = HudiTestProperties.minimalMap(); - properties.putAll(selection); - Assertions.assertThrows(IllegalArgumentException.class, - () -> HudiCatalogProperties.of(properties)); - } - } - - @Test - public void preservesGcsHmacAndS3Authentication() { - for (String provider : new String[] {"GCP", "S3"}) { - for (String prefix : new String[] {"gs.", "s3."}) { - Map properties = HudiTestProperties.minimalMap(); - properties.put("provider", provider); - properties.put("s3.endpoint", "https://storage.googleapis.com"); - properties.put(prefix + "access_key", "test-access-key"); - properties.put(prefix + "secret_key", "test-secret-key"); - Assertions.assertDoesNotThrow(() -> HudiCatalogProperties.of(properties)); - } - } - Map s3 = HudiTestProperties.minimalMap(); - s3.put("provider", "S3"); - s3.put("s3.role_arn", "arn:aws:iam::123456789012:role/test-role"); - Assertions.assertDoesNotThrow(() -> HudiCatalogProperties.of(s3)); - } - - @Test - public void preservesExplicitAnonymousGcsAccess() { - Map properties = HudiTestProperties.minimalMap(); - properties.put("provider", "GCP"); - properties.put("gs.credential_provider_type", "ANONYMOUS"); - Assertions.assertDoesNotThrow(() -> HudiCatalogProperties.of(properties)); - } } diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java index 9df7f85f732f15..18215a3858021b 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonCatalogProperties.java @@ -19,7 +19,6 @@ import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.metastore.spi.MetaStoreProviders; -import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import org.apache.doris.foundation.property.ConnectorProperty; @@ -121,12 +120,6 @@ private PaimonCatalogProperties(Map properties) { * overwrite a key, never remove one, so a rejected unknown key could not be repaired. */ public static PaimonCatalogProperties of(Map properties) { - // Reject unsupported native auth before creating clients, including catalog rebuilds. - // Resolve the mode centrally so GCS HMAC and existing S3 configurations remain supported. - if (GcsAuthResolver.resolve(properties).flatMap(auth -> auth.getNativeCredential()).isPresent()) { - throw new IllegalArgumentException("Native GCP authentication is not supported for Paimon yet; " - + "use GCS HMAC credentials instead."); - } PaimonCatalogProperties p = new PaimonCatalogProperties(properties); ConnectorPropertiesUtils.bindConnectorProperties(p, properties); p.flavor = p.catalogType.toLowerCase(Locale.ROOT); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java index a41d2520024cfa..1e97a092030cfd 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonCatalogPropertiesTest.java @@ -173,62 +173,4 @@ public void rawMapIsNotWritable() { Assertions.assertThrows(UnsupportedOperationException.class, () -> p.getRaw().put("k", "v")); } - - @Test - public void rejectsNativeGcpAuthenticationBeforeClientCreation() { - for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { - for (boolean impersonate : new boolean[] {false, true}) { - Map properties = props("warehouse", "s3a://bucket/warehouse"); - properties.put("provider", "GCP"); - properties.put("gs.credential_provider_type", mode); - if (impersonate) { - properties.put("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"); - } - IllegalArgumentException error = Assertions.assertThrows(IllegalArgumentException.class, - () -> PaimonCatalogProperties.of(properties)); - Assertions.assertTrue(error.getMessage().contains( - "Native GCP authentication is not supported for Paimon")); - } - } - } - - @Test - public void rejectsImplicitGcpAdcAndImpersonationOnly() { - for (Map selection : java.util.List.of( - Map.of("provider", "GCP"), - Map.of("warehouse", "gs://bucket/warehouse"), - Map.of("gs.endpoint", "https://storage.googleapis.com"), - Map.of("gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"))) { - Map properties = props("warehouse", "s3a://bucket/warehouse"); - properties.putAll(selection); - Assertions.assertThrows(IllegalArgumentException.class, - () -> PaimonCatalogProperties.of(properties)); - } - } - - @Test - public void preservesGcsHmacAndS3Authentication() { - for (String provider : new String[] {"GCP", "S3"}) { - for (String prefix : new String[] {"gs.", "s3."}) { - Map properties = props("warehouse", "s3a://bucket/warehouse"); - properties.put("provider", provider); - properties.put("s3.endpoint", "https://storage.googleapis.com"); - properties.put(prefix + "access_key", "test-access-key"); - properties.put(prefix + "secret_key", "test-secret-key"); - Assertions.assertDoesNotThrow(() -> PaimonCatalogProperties.of(properties)); - } - } - Map s3 = props("warehouse", "s3a://bucket/warehouse"); - s3.put("provider", "S3"); - s3.put("s3.role_arn", "arn:aws:iam::123456789012:role/test-role"); - Assertions.assertDoesNotThrow(() -> PaimonCatalogProperties.of(s3)); - } - - @Test - public void preservesExplicitAnonymousGcsAccess() { - Map properties = props("warehouse", "s3a://bucket/warehouse"); - properties.put("provider", "GCP"); - properties.put("gs.credential_provider_type", "ANONYMOUS"); - Assertions.assertDoesNotThrow(() -> PaimonCatalogProperties.of(properties)); - } } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java index a4a3287362d6ab..991cdffd535aae 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorValidatePropertiesTest.java @@ -215,21 +215,4 @@ public void dlfCatalogTypeValidatesLegacyAliasOnlyProperties() { "dlf.secret_key", "sk", "dlf.region", "cn-hangzhou")); } - - @Test - public void rejectsNativeGcpOnCreateAndAlter() { - Map nativeGcp = props("provider", "GCP", - "gs.credential_provider_type", "COMPUTE_ENGINE", - "gs.impersonation_service_account", "reader@example.iam.gserviceaccount.com"); - Map properties = props("warehouse", "s3a://bucket/warehouse"); - properties.putAll(nativeGcp); - IllegalArgumentException createError = Assertions.assertThrows(IllegalArgumentException.class, - () -> PROVIDER.validateProperties(properties)); - Assertions.assertTrue(createError.getMessage().contains( - "Native GCP authentication is not supported for Paimon")); - IllegalArgumentException alterError = Assertions.assertThrows(IllegalArgumentException.class, - () -> PROVIDER.validatePropertiesForUpdate(props("warehouse", "s3a://bucket/warehouse"), nativeGcp)); - Assertions.assertTrue(alterError.getMessage().contains( - "Native GCP authentication is not supported for Paimon")); - } } From 4ef6ccfa5744607cf09e7369d31ec344fcf2ff42 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Sat, 10 Oct 2026 10:47:11 +0800 Subject: [PATCH 23/47] [fix](fe) Preserve S3 selection and recognize GCS endpoint hosts ### What problem does this PR solve? Related PR: #68540 Problem Summary: Direct filesystem binding ignored provider=S3 at Google storage endpoints and could silently select GCS ADC. GCS endpoint inference also compared raw suffixes, missing explicit ports, trailing slashes and regional hosts. Honor explicit S3 in typed provider selection, prevent duplicate GCS/MinIO bindings, and share URI host recognition with the existing native OAuth endpoint validator. ### Release note Explicit S3 configurations retain AWS authentication. GCS endpoint inference now recognizes HTTPS port 443, trailing slashes and supported regional hosts while preserving native OAuth destination restrictions. ### Check List (For Author) - Test: Added resolver/provider tests and StorageAdapter primary/all binding regression tests. Checkstyle passed for API, S3 base, S3, GCS, MinIO and FE core; git diff --check passed. Unit tests were not run because the local FE prerequisites (protoc and JDK 17) are unavailable. - Behavior changed: Yes; preserve explicit S3 authentication and correctly infer supported GCS endpoints. - Does this need documentation: No --- .../apache/doris/fs/GcsGuessRoutingTest.java | 108 ++++++++++++++++++ .../filesystem/auth/GcsAuthResolver.java | 2 +- .../doris/filesystem/auth/GcsEndpoint.java | 73 ++++++++++++ .../filesystem/auth/GcsAuthResolverTest.java | 30 +++++ .../apache/doris/filesystem/gcs/GcpAuth.java | 32 +----- .../filesystem/gcs/GcsFileSystemProvider.java | 2 +- .../gcs/GcsFileSystemProviderTest.java | 13 +++ .../minio/MinioFileSystemProvider.java | 4 + .../minio/MinioFileSystemProviderTest.java | 12 ++ .../doris/filesystem/s3/S3CompatSignals.java | 2 +- .../filesystem/s3/S3FileSystemProvider.java | 2 +- .../s3/S3FileSystemProviderTest.java | 8 ++ 12 files changed, 255 insertions(+), 33 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java new file mode 100644 index 00000000000000..4acbe26f70add8 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.fs; + +import org.apache.doris.datasource.storage.StorageAdapter; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +public class GcsGuessRoutingTest { + private static final String[] ENDPOINTS = { + "https://storage.googleapis.com", "https://storage.googleapis.com:443", + "https://storage.googleapis.com/", "storage.googleapis.com:443/", + "https://storage.us-east1.rep.googleapis.com", "https://us-east1-storage.googleapis.com"}; + + @Test + public void testExplicitS3KeepsAwsAuthenticationInPrimaryAndAllBindings() { + for (String endpoint : ENDPOINTS) { + for (String provider : new String[] {"S3", "s3"}) { + Map props = properties(endpoint); + props.put("provider", provider); + assertS3Bindings(props, "DEFAULT"); + props.put("s3.access_key", "ak"); + props.put("s3.secret_key", "sk"); + assertS3Bindings(props, "DEFAULT"); + props.remove("s3.access_key"); + props.remove("s3.secret_key"); + props.put("s3.credentials_provider_type", "ANONYMOUS"); + assertS3Bindings(props, "ANONYMOUS"); + } + } + } + + @Test + public void testInferredGcsEndpointsBindNativeAndHmacWithoutMinio() { + for (String endpoint : ENDPOINTS) { + Map props = properties(endpoint); + for (StorageAdapter adapter : bindings(props, "GCS")) { + Assertions.assertEquals("DEFAULT", + adapter.getBackendConfigProperties().get("gs.credential_provider_type")); + } + props.put("s3.access_key", "ak"); + props.put("s3.secret_key", "sk"); + for (StorageAdapter adapter : bindings(props, "GCS")) { + Assertions.assertFalse(adapter.getBackendConfigProperties().containsKey("gs.credential_provider_type")); + Assertions.assertEquals("ak", adapter.getBackendConfigProperties().get("AWS_ACCESS_KEY")); + } + } + } + + @Test + public void testInferredNativeGcsStillValidatesEndpoint() { + for (String endpoint : new String[] {"http://storage.googleapis.com", + "https://storage.googleapis.com:8443", "https://storage.googleapis.com/bucket", + "https://storage.googleapis.com?query=value", "https://bucket.storage.googleapis.com"}) { + Map props = properties(endpoint); + Assertions.assertThrows(IllegalArgumentException.class, () -> StorageAdapter.of(props), endpoint); + Assertions.assertThrows(IllegalArgumentException.class, () -> StorageAdapter.ofAll(props), endpoint); + } + } + + private static Map properties(String endpoint) { + Map props = new HashMap<>(); + props.put("s3.endpoint", endpoint); + props.put("s3.region", "us-east1"); + props.put("uri", "s3://bucket/key"); + return props; + } + + private static void assertS3Bindings(Map props, String mode) { + for (StorageAdapter adapter : bindings(props, "S3")) { + Map backend = adapter.getBackendConfigProperties(); + Assertions.assertEquals(mode, backend.get("AWS_CREDENTIALS_PROVIDER_TYPE")); + Assertions.assertFalse(backend.containsKey("gs.credential_provider_type")); + } + } + + private static List bindings(Map props, String expectedProvider) { + StorageAdapter primary = StorageAdapter.of(props); + Assertions.assertEquals(expectedProvider, primary.getSpiProperties().providerName()); + // bindAll also supplies the default HDFS filesystem for catalog callers. + List objectBindings = StorageAdapter.ofAll(props).stream() + .filter(adapter -> !"HDFS".equals(adapter.getStorageName())).collect(Collectors.toList()); + Assertions.assertEquals(1, objectBindings.size(), props.toString()); + Assertions.assertEquals(expectedProvider, objectBindings.get(0).getSpiProperties().providerName()); + return List.of(primary, objectBindings.get(0)); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java index cb01d70cde36cb..a61542d3499c56 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java @@ -123,7 +123,7 @@ && isNotBlank(value)) { return true; } if (("s3.endpoint".equals(key) || "aws_endpoint".equals(key) || "endpoint".equals(key)) - && value != null && value.toLowerCase(Locale.ROOT).endsWith("storage.googleapis.com")) { + && value != null && GcsEndpoint.isGcsEndpoint(value)) { return true; } } diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java new file mode 100644 index 00000000000000..9eb84772f2789f --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import java.net.URI; +import java.util.regex.Pattern; + +/** Shared GCS host recognition and native OAuth endpoint validation. */ +public final class GcsEndpoint { + // Endpoint overrides must be service-base hosts; SDK resolvers add the bucket. + private static final Pattern STORAGE_BASE_HOST = Pattern.compile( + "(?:storage\\.googleapis\\.com|[a-z0-9-]+-storage\\.googleapis\\.com|" + + "storage\\.[a-z0-9-]+\\.rep\\.googleapis\\.com)", Pattern.CASE_INSENSITIVE); + // Resolved virtual-hosted requests legitimately contain the bucket prefix. + private static final Pattern STORAGE_HOST = Pattern.compile( + "(?:[a-z0-9-]+\\.)*" + STORAGE_BASE_HOST.pattern(), Pattern.CASE_INSENSITIVE); + + private GcsEndpoint() { + } + + /** Identifies GCS by host; authentication-specific restrictions are checked after binding. */ + public static boolean isGcsEndpoint(String endpoint) { + try { + String host = parse(endpoint).getHost(); + return host != null && STORAGE_HOST.matcher(host).matches(); + } catch (IllegalArgumentException e) { + // A malformed URI cannot identify a filesystem provider. + return false; + } + } + + private static URI parse(String endpoint) { + return URI.create(endpoint.contains("://") ? endpoint : "https://" + endpoint); + } + + public static URI validateEndpoint(String endpoint) { + URI uri = parse(endpoint); + validateRequestUri(uri); + if (!STORAGE_BASE_HOST.matcher(uri.getHost()).matches()) { + throw new IllegalArgumentException( + "Native GCP OAuth endpoint must be a service-base host without a bucket"); + } + if (uri.getRawQuery() != null || (!uri.getRawPath().isEmpty() && !"/".equals(uri.getRawPath()))) { + throw new IllegalArgumentException("Native GCP OAuth endpoint must not contain a path or query"); + } + return uri; + } + + public static void validateRequestUri(URI uri) { + if (!"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null + || !STORAGE_HOST.matcher(uri.getHost()).matches() + || (uri.getPort() != -1 && uri.getPort() != 443) + || uri.getRawUserInfo() != null || uri.getRawFragment() != null) { + throw new IllegalArgumentException( + "Native GCP OAuth requires a trusted Google Cloud Storage HTTPS endpoint on port 443"); + } + } +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java index 5e7bbfdbd0f80d..b2363a362a2332 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java @@ -119,4 +119,34 @@ public void testOtherProvidersAndConflictingNativeProperties() { props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, ""); Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); } + + @Test + public void testEndpointAliasesResolveGcsByUriHost() { + for (String key : new String[] {"s3.endpoint", "AWS_ENDPOINT", "endpoint", "ENDPOINT"}) { + for (String endpoint : new String[] {"storage.googleapis.com", "storage.googleapis.com:443", + "https://storage.googleapis.com/", "https://storage.googleapis.com:443/", + "HTTPS://STORAGE.GOOGLEAPIS.COM", "https://storage.us-east1.rep.googleapis.com", + "https://us-east1-storage.googleapis.com", "https://bucket.storage.googleapis.com"}) { + Map props = new HashMap<>(); + props.put(key, endpoint); + Assertions.assertTrue(GcsAuthResolver.guessIsGcs(props), endpoint); + Assertions.assertEquals(GcsAuth.Mode.ADC, GcsAuthResolver.resolve(props).orElseThrow().getMode()); + props.put("s3.access_key", "ak"); + props.put("s3.secret_key", "sk"); + Assertions.assertEquals(GcsAuth.Mode.HMAC, GcsAuthResolver.resolve(props).orElseThrow().getMode()); + } + } + } + + @Test + public void testEndpointInferenceRejectsForeignAndMalformedHosts() { + for (String endpoint : new String[] {"https://notstorage.googleapis.com", + "https://storage.googleapis.com.example.com", "https://example.com/storage.googleapis.com", + "https://storage.googleapis.com@example.com", "https://[invalid", ""}) { + Map props = Map.of("s3.endpoint", endpoint); + Assertions.assertFalse(GcsAuthResolver.guessIsGcs(props), endpoint); + Assertions.assertFalse(GcsAuthResolver.resolve(props).isPresent(), endpoint); + } + } + } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java index dbcf265588966a..87e6c776a816c7 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java @@ -18,6 +18,7 @@ package org.apache.doris.filesystem.gcs; import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.GcsEndpoint; import com.google.auth.oauth2.ComputeEngineCredentials; import com.google.auth.oauth2.GoogleCredentials; @@ -45,7 +46,6 @@ import java.util.Collections; import java.util.List; import java.util.Map; -import java.util.regex.Pattern; /** Creates GCP OAuth2 credentials and adapts them to the S3-compatible client. */ public final class GcpAuth { @@ -53,13 +53,6 @@ public final class GcpAuth { "https://www.googleapis.com/auth/devstorage.read_write"; private static final String CLOUD_PLATFORM_SCOPE = "https://www.googleapis.com/auth/cloud-platform"; private static final int IMPERSONATION_LIFETIME_SECONDS = 3600; - // Endpoint overrides must be service-base hosts; SDK resolvers add the bucket. - private static final Pattern STORAGE_BASE_HOST = Pattern.compile( - "(?:storage\\.googleapis\\.com|[a-z0-9-]+-storage\\.googleapis\\.com|" - + "storage\\.[a-z0-9-]+\\.rep\\.googleapis\\.com)", Pattern.CASE_INSENSITIVE); - // Resolved virtual-hosted requests legitimately contain the bucket prefix. - private static final Pattern STORAGE_HOST = Pattern.compile( - "(?:[a-z0-9-]+\\.)*" + STORAGE_BASE_HOST.pattern(), Pattern.CASE_INSENSITIVE); private GcpAuth() { } @@ -91,26 +84,7 @@ private static URI endpoint(GcsFileSystemProperties properties) { } static URI validateEndpoint(String endpoint) { - URI uri = URI.create(endpoint.contains("://") ? endpoint : "https://" + endpoint); - validateRequestUri(uri); - if (!STORAGE_BASE_HOST.matcher(uri.getHost()).matches()) { - throw new IllegalArgumentException( - "Native GCP OAuth endpoint must be a service-base host without a bucket"); - } - if (uri.getRawQuery() != null || (!uri.getRawPath().isEmpty() && !"/".equals(uri.getRawPath()))) { - throw new IllegalArgumentException("Native GCP OAuth endpoint must not contain a path or query"); - } - return uri; - } - - private static void validateRequestUri(URI uri) { - if (!"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null - || !STORAGE_HOST.matcher(uri.getHost()).matches() - || (uri.getPort() != -1 && uri.getPort() != 443) - || uri.getRawUserInfo() != null || uri.getRawFragment() != null) { - throw new IllegalArgumentException( - "Native GCP OAuth requires a trusted Google Cloud Storage HTTPS endpoint on port 443"); - } + return GcsEndpoint.validateEndpoint(endpoint); } private static S3Configuration serviceConfiguration(GcsFileSystemProperties properties) { @@ -161,7 +135,7 @@ public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) { try { // Validate the resolved request host before obtaining or attaching process credentials. - validateRequestUri(context.httpRequest().getUri()); + GcsEndpoint.validateRequestUri(context.httpRequest().getUri()); Map> metadata = credentials.getRequestMetadata(); List authorization = metadata.get("Authorization"); if (authorization == null || authorization.isEmpty() diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index 9f7f3c1e050019..716f9a560b2dfe 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -83,7 +83,7 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { - return S3CompatSignals.guessIsGcs(properties); + return !S3CompatSignals.hasExplicitS3Request(properties) && S3CompatSignals.guessIsGcs(properties); } @Override diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java index 6a08156b9ace4c..cf1891e809e3d7 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java @@ -199,4 +199,17 @@ void nativeOAuthRejectsAwsPresigningBeforeCredentialLoading() { "AWS_CREDENTIALS_PROVIDER_TYPE", "ANONYMOUS")), props); Assertions.assertThrows(UnsupportedOperationException.class, () -> storage.getPresignedUrl("key")); } + + @Test + void supportsGuess_respectsExplicitS3AndRecognizesRegionalEndpoints() { + Map props = new HashMap<>(); + for (String endpoint : new String[] {"https://storage.googleapis.com:443/", + "https://storage.us-east1.rep.googleapis.com"}) { + props.put("s3.endpoint", endpoint); + Assertions.assertTrue(provider.supportsGuess(props)); + props.put("provider", "S3"); + Assertions.assertFalse(provider.supportsGuess(props)); + props.remove("provider"); + } + } } diff --git a/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java index f08e741dd39b2f..245f999117472b 100644 --- a/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java @@ -89,6 +89,10 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { + // bindAll consults every provider, even after S3 or GCS has matched. + if (S3CompatSignals.hasExplicitS3Request(properties) || S3CompatSignals.guessIsGcs(properties)) { + return false; + } // Verbatim port of fe-core MinioProperties.guessIsMe: MinIO is the "any other // S3-compatible" fallback — it claims the map iff none of Azure/COS/OSS/S3 would claim // it AND at least one identifier key is present. The sibling guesses are replicated diff --git a/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java index 1058dedc7fab0b..eab1db44030cbb 100644 --- a/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java @@ -195,4 +195,16 @@ void sensitivePropertyKeys_coverSecretAliasesButNotAccessKey() { void name_isMinio() { Assertions.assertEquals("MINIO", provider.name()); } + + @Test + void supportsGuess_doesNotClaimExplicitS3OrRegionalGcs() { + for (String endpoint : new String[] {"https://storage.googleapis.com:443/", + "https://storage.us-east1.rep.googleapis.com", "https://us-east1-storage.googleapis.com"}) { + Map props = new HashMap<>(); + props.put("s3.endpoint", endpoint); + Assertions.assertFalse(provider.supportsGuess(props)); + props.put("provider", "S3"); + Assertions.assertFalse(provider.supportsGuess(props)); + } + } } diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java index 9f4ba05db6a2ee..b158f88a123462 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java @@ -173,7 +173,7 @@ public static boolean guessAllowed(Map properties) { /** * Port of {@code GCSProperties.guessIsMe}: a non-blank {@code gs.endpoint}, or any endpoint alias - * whose value ends with {@code storage.googleapis.com}. + * whose URI host identifies a GCS global, regional, or virtual-hosted endpoint. * *

Deviation: legacy compares {@code key.toLowerCase()} against a set holding the un-lowercased * {@code "AWS_ENDPOINT"}/{@code "ENDPOINT"}, so those two aliases can never match there. Here the diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java index 18710ecbfad0ed..e4a8d3b9c39656 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java @@ -110,7 +110,7 @@ public FileSystem create(S3FileSystemProperties properties) throws IOException { @Override public boolean supportsExplicit(Map properties) { - return Boolean.parseBoolean(properties.getOrDefault(FS_S3_SUPPORT, "false")); + return S3CompatSignals.hasExplicitS3Request(properties); } @Override diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java index 2c3870384857d4..360a41ee838c78 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java @@ -252,4 +252,12 @@ void sensitivePropertyKeysCoverSecretsButNotAccessKey() { Assertions.assertFalse(keys.contains("s3.access_key"), keys.toString()); Assertions.assertFalse(keys.contains("AWS_ACCESS_KEY"), keys.toString()); } + + @Test + void supportsExplicit_preservesS3DefaultChainAtGcsEndpoint() { + Map props = Map.of("provider", "s3", + "s3.endpoint", "https://storage.googleapis.com", "s3.region", "us-east1"); + Assertions.assertTrue(provider.supportsExplicit(props)); + Assertions.assertEquals("DEFAULT", provider.bind(props).toFileSystemKv().get("AWS_CREDENTIALS_PROVIDER_TYPE")); + } } From 30f208fc0e5c74fd4160892cd2989f1d17a1dc6f Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Sat, 10 Oct 2026 11:31:47 +0800 Subject: [PATCH 24/47] [fix](storage) Honor GCP signing CA and preserve Iceberg FileIO selection ### What problem does this PR solve? Related PR: #68540 Problem Summary: IAM signBlob omitted the CA bundle used by the object client and OAuth provider, so private-CA deployments could upload error logs but fail to sign them. Pass the selected CA through the signed-URL callback into HttpClient. Native GCP catalog setup also forced S3FileIO for every location, overwriting explicit io-impl values and misrouting HDFS locations to GCS. Preserve explicit FileIO choices and default to Iceberg's scheme-aware ResolvingFileIO, retaining the GCP S3 client factory for compatible S3 aliases and the native Hadoop GCS configuration for gs locations. ### Release note GCP IAM signing honors the configured CA bundle. Native GCP Iceberg catalogs preserve explicit FileIO configuration and route mixed filesystem locations by scheme. ### Check List (For Author) - Test: Added CA propagation and explicit/mixed-scheme FileIO regression tests. Checkstyle, clang-format 16 check, connector import gate, build hygiene and git diff --check passed. Unit tests and clang-tidy were not run: FE protoc/JDK 17 and BE build artifacts are unavailable locally. - Behavior changed: Yes; fix signing with private CAs and preserve Iceberg filesystem routing. - Does this need documentation: No --- be/src/io/fs/gcs_signed_url_provider.cpp | 12 ++-- be/src/io/fs/gcs_signed_url_provider.h | 1 + be/src/service/http/http_client.cpp | 10 +++ be/src/service/http/http_client.h | 4 +- be/src/util/s3_util.cpp | 5 +- .../io/fs/gcs_signed_url_provider_test.cpp | 29 ++++++++ .../iceberg/IcebergCatalogFactory.java | 5 +- .../iceberg/IcebergCatalogFactoryTest.java | 66 ++++++++++++++++++- 8 files changed, 123 insertions(+), 9 deletions(-) diff --git a/be/src/io/fs/gcs_signed_url_provider.cpp b/be/src/io/fs/gcs_signed_url_provider.cpp index 79a040fa7523a6..85e08264f29710 100644 --- a/be/src/io/fs/gcs_signed_url_provider.cpp +++ b/be/src/io/fs/gcs_signed_url_provider.cpp @@ -85,9 +85,7 @@ std::string percent_encode(std::string_view value) { Status call_iam_sign_blob(std::string_view access_token, std::string_view service_account, std::string_view string_to_sign, int64_t request_timeout_ms, - std::string* signature) { - TEST_SYNC_POINT_RETURN_WITH_VALUE("GcsV4Signer::sign_blob", Status::OK(), access_token, - service_account, string_to_sign, signature); + const std::string& ca_cert_file_path, std::string* signature) { std::string encoded_payload; base64_encode(std::string(string_to_sign), &encoded_payload); std::string request_body = fmt::format(R"({{"payload":"{}"}})", encoded_payload); @@ -98,10 +96,15 @@ Status call_iam_sign_blob(std::string_view access_token, std::string_view servic // Keep the response body for non-2xx replies so IAM permission and // service-account errors are actionable to operators. RETURN_IF_ERROR(client.init(endpoint, false, HttpClient::AuthTokenMode::NONE)); + if (!ca_cert_file_path.empty()) { + RETURN_IF_ERROR(client.set_ca_cert_file(ca_cert_file_path)); + } client.set_authorization("Bearer " + std::string(access_token)); client.set_content_type("application/json"); client.set_timeout_ms(request_timeout_ms > 0 ? request_timeout_ms : 10000); + TEST_SYNC_POINT_RETURN_WITH_VALUE("GcsV4Signer::sign_blob", Status::OK(), access_token, + service_account, string_to_sign, signature); std::string response; RETURN_IF_ERROR(client.execute_post_request(request_body, &response)); @@ -170,7 +173,8 @@ Status generate_gcs_v4_signed_url(const GcsV4SignedUrlProviderOptions& options, std::chrono::system_clock::now(), [&](std::string_view string_to_sign) { std::string signature; auto status = call_iam_sign_blob(*token, signer_email, string_to_sign, - options.request_timeout_ms, &signature); + options.request_timeout_ms, + options.ca_cert_file_path, &signature); if (!status.ok()) { return GcsSignBlobResult {.error = status.to_string()}; } diff --git a/be/src/io/fs/gcs_signed_url_provider.h b/be/src/io/fs/gcs_signed_url_provider.h index c925e758042d09..872622059223bb 100644 --- a/be/src/io/fs/gcs_signed_url_provider.h +++ b/be/src/io/fs/gcs_signed_url_provider.h @@ -36,6 +36,7 @@ struct GcsV4SignedUrlProviderOptions { std::string key; int64_t expiration_secs = 0; int64_t request_timeout_ms = 10000; + std::string ca_cert_file_path; }; // Resolve the identity selected by ADC or explicit impersonation, without falling back diff --git a/be/src/service/http/http_client.cpp b/be/src/service/http/http_client.cpp index 59c5e95211ff6b..2ace3157069bd0 100644 --- a/be/src/service/http/http_client.cpp +++ b/be/src/service/http/http_client.cpp @@ -27,6 +27,7 @@ #include "common/cast_set.h" #include "common/config.h" #include "common/status.h" +#include "cpp/sync_point.h" #include "io/fs/local_file_system.h" #include "runtime/cluster_info.h" #include "runtime/exec_env.h" @@ -367,6 +368,15 @@ Status HttpClient::init(const std::string& url, bool set_fail_on_error, return Status::OK(); } +Status HttpClient::set_ca_cert_file(const std::string& path) { + auto code = curl_easy_setopt(_curl, CURLOPT_CAINFO, path.c_str()); + if (code != CURLE_OK) { + return Status::InternalError("failed to set CA certificate file: {}", _to_errmsg(code)); + } + TEST_SYNC_POINT_CALLBACK("HttpClient::set_ca_cert_file", &path); + return Status::OK(); +} + void HttpClient::set_method(HttpMethod method) { _method = method; switch (method) { diff --git a/be/src/service/http/http_client.h b/be/src/service/http/http_client.h index b79a9a6c14ce90..724d54ce680eff 100644 --- a/be/src/service/http/http_client.h +++ b/be/src/service/http/http_client.h @@ -86,7 +86,9 @@ class HttpClient { curl_easy_setopt(_curl, CURLOPT_COPYPOSTFIELDS, post_body.c_str()); } - // Currently, only fake SSL configurations are supported + // Override the trusted CA bundle after init(), which resets curl options. + Status set_ca_cert_file(const std::string& path); + void use_untrusted_ssl() { curl_easy_setopt(_curl, CURLOPT_SSL_VERIFYPEER, 0L); curl_easy_setopt(_curl, CURLOPT_SSL_VERIFYHOST, 0L); diff --git a/be/src/util/s3_util.cpp b/be/src/util/s3_util.cpp index 5904d399f645fe..2db85be3260dc0 100644 --- a/be/src/util/s3_util.cpp +++ b/be/src/util/s3_util.cpp @@ -458,7 +458,7 @@ Result> S3ClientFactory::_create_s3_client S3ObjStorageClient::SignedUrlGenerator signed_url_generator; if (const auto* credential = std::get_if(&s3_conf.credential)) { - signed_url_generator = [conf = s3_conf, credential = *credential, + signed_url_generator = [conf = s3_conf, credential = *credential, ca_cert_file_path, token_provider = std::move(gcs_signing_token_provider)]( const ObjStoragePath& path, int64_t expiration_secs) { std::string signed_url; @@ -467,7 +467,8 @@ Result> S3ClientFactory::_create_s3_client .bucket = path.bucket, .key = path.key, .expiration_secs = expiration_secs, - .request_timeout_ms = conf.request_timeout_ms}, + .request_timeout_ms = conf.request_timeout_ms, + .ca_cert_file_path = ca_cert_file_path}, credential, token_provider, &signed_url); if (!status.ok()) { LOG(WARNING) << "failed to generate GCS V4 signed URL: " << status; diff --git a/be/test/io/fs/gcs_signed_url_provider_test.cpp b/be/test/io/fs/gcs_signed_url_provider_test.cpp index 98b6f7b134a593..6d4e17a72c991c 100644 --- a/be/test/io/fs/gcs_signed_url_provider_test.cpp +++ b/be/test/io/fs/gcs_signed_url_provider_test.cpp @@ -47,6 +47,7 @@ class GcsSignedUrlProviderTest : public testing::Test { protected: void TearDown() override { SyncPoint::get_instance()->clear_call_back("GcsV4Signer::sign_blob"); + SyncPoint::get_instance()->clear_call_back("HttpClient::set_ca_cert_file"); SyncPoint::get_instance()->disable_processing(); } }; @@ -81,6 +82,34 @@ TEST_F(GcsSignedUrlProviderTest, DefaultSignsErrorLogWithResolvedIdentity) { EXPECT_NE(url.find("X-Goog-Signature="), std::string::npos); } +TEST_F(GcsSignedUrlProviderTest, SignBlobUsesConfiguredCaBundle) { + auto tokens = std::make_shared( + std::make_shared("vm@my-project.iam.gserviceaccount.com")); + GcsV4SignedUrlProviderOptions options {.endpoint = "storage.googleapis.com", + .bucket = "bucket", + .key = "load-errors/error.log", + .expiration_secs = 300, + .ca_cert_file_path = "/custom/certs/proxy-ca.pem"}; + bool called = false; + SyncPoint::get_instance()->set_call_back( + "HttpClient::set_ca_cert_file", [&](std::vector&& args) { + called = true; + EXPECT_EQ(*try_any_cast(args[0]), options.ca_cert_file_path); + }); + SyncPoint::get_instance()->set_call_back( + "GcsV4Signer::sign_blob", [&](std::vector&& args) { + // Exercise HTTP client setup but avoid sending a live IAM request. + *try_any_cast(args[3]) = "test-signature"; + try_any_cast_ret(args)->second = true; + }); + SyncPoint::get_instance()->enable_processing(); + std::string url; + auto status = generate_gcs_v4_signed_url(options, GcpCredentialConfig {}, tokens, &url); + ASSERT_TRUE(status.ok()) << status; + EXPECT_TRUE(called); + EXPECT_NE(url.find("X-Goog-Signature="), std::string::npos); +} + TEST_F(GcsSignedUrlProviderTest, DefaultAndComputeUseTheSelectedAccount) { // Represents ADC selecting either the VM account or a service-account file. // Both must use the account attached to the token, without a separate metadata lookup. diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index 662e9a287a3e2d..a75b02d8ea41ea 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -445,7 +445,10 @@ public static Map buildCatalogProperties(IcebergCatalogPropertie opts.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); putIfNotBlank(opts, GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, credential.getImpersonationServiceAccount()); - opts.put(CatalogProperties.FILE_IO_IMPL, "org.apache.iceberg.aws.s3.S3FileIO"); + // Keep explicit FileIO choices. By default, S3 aliases use the GCP client factory, + // while gs/HDFS paths use HadoopFileIO with the native fs.gs.* configuration + // (the plugin does not bundle iceberg-gcp's GCSFileIO). + opts.putIfAbsent(CatalogProperties.FILE_IO_IMPL, "org.apache.iceberg.io.ResolvingFileIO"); opts.put(S3FileIOProperties.CLIENT_FACTORY, GcpS3FileIOAwsClientFactory.class.getName()); }); })); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index 89d6563153379d..10b8cb1828b8bd 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -28,9 +28,17 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.RawLocalFileSystem; import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.iceberg.CatalogUtil; import org.apache.iceberg.aws.AwsClientProperties; import org.apache.iceberg.aws.s3.S3FileIOProperties; +import org.apache.iceberg.aws.s3.S3InputFile; +import org.apache.iceberg.hadoop.HadoopFileIO; +import org.apache.iceberg.hadoop.HadoopInputFile; +import org.apache.iceberg.io.FileIO; +import org.apache.iceberg.io.InputFile; +import org.apache.iceberg.io.SeekableInputStream; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -41,6 +49,8 @@ import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider; import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; import java.util.Arrays; import java.util.Collections; import java.util.HashMap; @@ -1062,7 +1072,7 @@ void nativeGcpCatalogUsesBearerFileIoAndNativeHadoopConfiguration() { properties.put("iceberg.jdbc.uri", "jdbc:postgresql://localhost/iceberg"); Map options = IcebergCatalogFactory.buildCatalogProperties( IcebergCatalogProperties.of(properties), Optional.of(storage)); - Assertions.assertEquals("org.apache.iceberg.aws.s3.S3FileIO", options.get("io-impl")); + Assertions.assertEquals("org.apache.iceberg.io.ResolvingFileIO", options.get("io-impl")); Assertions.assertEquals(GcpS3FileIOAwsClientFactory.class.getName(), options.get(S3FileIOProperties.CLIENT_FACTORY)); Assertions.assertEquals(source, options.get("gs.credential_provider_type")); @@ -1075,6 +1085,60 @@ void nativeGcpCatalogUsesBearerFileIoAndNativeHadoopConfiguration() { } } + @Test + void nativeGcpCatalogPreservesExplicitFileIo(@TempDir java.nio.file.Path temp) throws Exception { + Map raw = new HashMap<>(Map.of("provider", "GCP", + "warehouse", "gs://bucket/warehouse", "iceberg.catalog.type", "rest", + "uri", "http://localhost:8181", "gs.credential_provider_type", "COMPUTE_ENGINE", + "io-impl", HadoopFileIO.class.getName())); + GcsFileSystemProperties storage = GcsFileSystemProperties.of(raw); + Map options = IcebergCatalogFactory.buildCatalogProperties( + IcebergCatalogProperties.of(raw), Optional.of(storage)); + Assertions.assertEquals(HadoopFileIO.class.getName(), options.get("io-impl")); + Configuration conf = IcebergCatalogFactory.buildHadoopConfiguration(raw, + storage.toHadoopConfigurationMap()); + java.nio.file.Path metadata = temp.resolve("metadata.json"); + Files.writeString(metadata, "metadata", StandardCharsets.UTF_8); + try (FileIO io = CatalogUtil.loadFileIO(options.get("io-impl"), options, conf); + SeekableInputStream in = io.newInputFile(metadata.toUri().toString()).newStream()) { + Assertions.assertEquals("metadata", new String(in.readAllBytes(), StandardCharsets.UTF_8)); + } + } + + @Test + void nativeGcpRestCatalogRoutesMixedLocationsByScheme() throws Exception { + Map raw = Map.of("provider", "GCP", "warehouse", "gs://bucket/warehouse", + "iceberg.catalog.type", "rest", "uri", "http://localhost:8181", + "gs.credential_provider_type", "COMPUTE_ENGINE", + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"); + GcsFileSystemProperties storage = GcsFileSystemProperties.of(raw); + Map options = IcebergCatalogFactory.buildCatalogProperties( + IcebergCatalogProperties.of(raw), Optional.of(storage)); + Configuration conf = IcebergCatalogFactory.buildHadoopConfiguration(raw, + storage.toHadoopConfigurationMap()); + // Replace the Hadoop transports only, so the real FileIO dispatch runs without cloud/HDFS services. + for (String scheme : List.of("gs", "hdfs")) { + conf.set("fs." + scheme + ".impl", RawLocalFileSystem.class.getName()); + conf.setBoolean("fs." + scheme + ".impl.disable.cache", true); + } + try (FileIO io = CatalogUtil.loadFileIO(options.get("io-impl"), options, conf)) { + for (String location : List.of("gs://bucket/metadata.json", "hdfs://namenode/metadata.json")) { + InputFile input = io.newInputFile(location, 10); + Assertions.assertInstanceOf(HadoopInputFile.class, input, location); + HadoopInputFile hadoopInput = (HadoopInputFile) input; + try (FileSystem fs = hadoopInput.getFileSystem()) { + Assertions.assertInstanceOf(RawLocalFileSystem.class, fs); + Assertions.assertEquals("COMPUTE_ENGINE", fs.getConf().get("fs.gs.auth.type")); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", + fs.getConf().get("fs.gs.auth.impersonation.service.account")); + } + } + for (String scheme : List.of("s3", "s3a")) { + Assertions.assertInstanceOf(S3InputFile.class, io.newInputFile(scheme + "://bucket/metadata.json", 10)); + } + } + } + @Test void nativeGcpWarehouseAliasesSelectGoogleHadoopFileSystem() throws Exception { for (String scheme : List.of("gs", "s3", "s3a", "S3A")) { From 1b99a05afe85fb88b0f7191d21cfab0ab4170604 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Sat, 10 Oct 2026 12:09:24 +0800 Subject: [PATCH 25/47] [refactor](fe) isolate GCP authentication behind filesystem SPI ### What problem does this PR solve? Related PR: #68540 Problem Summary: Keep provider-specific GCS normalization, authentication parsing, credential patch rules and property codecs in filesystem plugins. Move GCP value and parsing classes from the public API into a shared implementation library and remove the core's Hadoop GCS connector dependency. Core consumes a generic immutable authentication descriptor and only translates native wire credentials. ### Release note Filesystem plugin API advances to 2.0 and connector plugin API to 13.0. Rebuild plugins with the matching FE. Existing native GCP, HMAC and S3 property and wire formats remain unchanged; Paimon and Hudi remain outside this feature. ### Check List (For Author) - Test: Checkstyle passed for all eight affected modules; connector import and core metadata-funnel gates passed; git diff --check and POM XML checks passed. - Test: Added provider normalization, credential patch/default, round-trip, HMAC and anonymous cases; retained and relocated auth resolver tests. Updated the filesystem API surface baseline. Compilation and unit tests were not run because thirdparty/installed/bin/protoc is missing in this checkout. - Behavior changed: Implementation ownership and plugin ABI versions only. - Does this need documentation: No new user-facing configuration. - Self-review: No new threads, locks, network calls, logging or lifecycle-owned resources. Resource mutation synchronization and publication remain intact. CREATE/ALTER, BE Thrift, Cloud protobuf and SHOW CREATE paths share the plugin rules; explicit empty ALTER credentials remain distinct from omitted fields. Persistence and FE/BE wire formats do not change. Plugin version gates reject incompatible binaries. No additional observability is needed for this refactor. --- .../iceberg/IcebergCatalogFactory.java | 6 +- .../spi/ConnectorPluginSurfaceTest.java | 2 +- fe/fe-connector/pom.xml | 2 +- fe/fe-core/pom.xml | 5 -- .../org/apache/doris/catalog/S3Resource.java | 83 ++++--------------- .../apache/doris/catalog/S3StorageVault.java | 6 +- .../apache/doris/catalog/StorageVaultMgr.java | 23 +---- .../cloud/storage/ObjectInfoAdapter.java | 6 +- .../storage/auth/GcpCredentialAdapter.java | 49 ++++++----- .../storage/auth/ObjCredentialFactory.java | 46 +++------- .../storage/CloudObjectStoreAdapter.java | 18 ++-- .../datasource/storage/S3ResourceCompat.java | 2 +- .../datasource/storage/S3ThriftAdapter.java | 11 ++- .../datasource/storage/StorageAdapter.java | 46 +++++++++- .../ShowCreateStorageVaultCommand.java | 9 +- .../analysis/StorageDescPersistTest.java | 5 +- .../apache/doris/catalog/S3ResourceTest.java | 27 +++--- .../cloud/catalog/S3StorageVaultTest.java | 15 ++-- .../nereids/trees/plans/OutFileTest.java | 10 +-- .../plans/commands/ExportCommandTest.java | 9 +- .../trees/plans/commands/LoadCommandTest.java | 7 +- .../ShowCreateStorageVaultCommandTest.java | 9 +- .../ExternalFileTableValuedFunctionTest.java | 11 ++- .../auth/ObjectStorageAuthentication.java | 63 ++++++++++++++ .../fe-filesystem-gcs-common/pom.xml | 42 ++++++++++ .../filesystem/gcs}/auth/GcpCredential.java | 2 +- .../gcs}/auth/GcpCredentialProviderType.java | 2 +- .../doris/filesystem/gcs}/auth/GcsAuth.java | 2 +- .../filesystem/gcs}/auth/GcsAuthResolver.java | 2 +- .../filesystem/gcs}/auth/GcsEndpoint.java | 2 +- .../gcs}/auth/GcsAuthResolverTest.java | 2 +- .../apache/doris/filesystem/gcs/GcpAuth.java | 4 +- .../gcs/GcsFileSystemProperties.java | 8 +- .../filesystem/gcs/GcsFileSystemProvider.java | 81 +++++++++++++++++- .../doris/filesystem/gcs/GcpAuthTest.java | 4 +- .../gcs/GcsFileSystemProviderTest.java | 67 +++++++++++++++ .../fe-filesystem-s3-base/pom.xml | 5 ++ .../doris/filesystem/s3/S3CompatSignals.java | 2 +- .../filesystem/s3/S3FileSystemProperties.java | 4 +- .../filesystem/spi/FileSystemProvider.java | 32 +++++++ .../spi/FileSystemPluginSurfaceTest.java | 1 + .../resources/filesystem-plugin-surface.txt | 10 +++ fe/fe-filesystem/pom.xml | 3 +- 43 files changed, 497 insertions(+), 248 deletions(-) create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java create mode 100644 fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml rename fe/fe-filesystem/{fe-filesystem-api/src/main/java/org/apache/doris/filesystem => fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs}/auth/GcpCredential.java (98%) rename fe/fe-filesystem/{fe-filesystem-api/src/main/java/org/apache/doris/filesystem => fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs}/auth/GcpCredentialProviderType.java (96%) rename fe/fe-filesystem/{fe-filesystem-api/src/main/java/org/apache/doris/filesystem => fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs}/auth/GcsAuth.java (97%) rename fe/fe-filesystem/{fe-filesystem-api/src/main/java/org/apache/doris/filesystem => fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs}/auth/GcsAuthResolver.java (99%) rename fe/fe-filesystem/{fe-filesystem-api/src/main/java/org/apache/doris/filesystem => fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs}/auth/GcsEndpoint.java (98%) rename fe/fe-filesystem/{fe-filesystem-api/src/test/java/org/apache/doris/filesystem => fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs}/auth/GcsAuthResolverTest.java (99%) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index a75b02d8ea41ea..ec2f177e0723f4 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -22,9 +22,9 @@ import org.apache.doris.connector.metastore.iceberg.jdbc.IcebergJdbcMetaStoreProperties; import org.apache.doris.connector.metastore.iceberg.rest.IcebergRestMetaStoreProperties; import org.apache.doris.connector.spi.DorisConnectorException; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcsAuth; -import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.filesystem.gcs.auth.GcpCredential; +import org.apache.doris.filesystem.gcs.auth.GcsAuth; +import org.apache.doris.filesystem.gcs.auth.GcsAuthResolver; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.filesystem.properties.StorageProperties; diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index a2b20ae311d973..bd90c101d88cc2 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -89,7 +89,7 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException } // Major 12 adds the SUPPORTS_FIELD_ID_ACCESS_PATH and SUPPORTS_SYS_TABLE_NESTED_COLUMN_PRUNE // capabilities: a plugin naming either constant cannot link against an older FE. - Assertions.assertEquals("12.0", version.getProperty("api.version")); + Assertions.assertEquals("13.0", version.getProperty("api.version")); } /** Root entry points plus provider/handle types returned to connector plugins. */ diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index 121868e3dd830f..9bf53c1050bfa3 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -55,7 +55,7 @@ under the License. of the latter two means bumping this property as well (and fe-extension-spi means bumping all five families). --> - 12.0 + 13.0 diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index e0ac341c64511e..656b9847c76147 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -457,11 +457,6 @@ under the License. commons-codec commons-codec - - com.google.cloud.bigdataoss - gcs-connector - shaded - org.apache.hadoop hadoop-azure diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index 03ce805b250f8b..dca93dae845dd5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -23,9 +23,8 @@ import org.apache.doris.common.util.DatasourcePrintableMap; import org.apache.doris.common.util.S3Util; import org.apache.doris.datasource.storage.S3ResourceCompat; +import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.filesystem.UploadPartResult; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcsAuthResolver; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.filesystem.spi.ObjStorage; import org.apache.doris.filesystem.spi.RequestBody; @@ -37,7 +36,6 @@ import com.google.common.collect.Lists; import com.google.common.collect.Maps; import com.google.gson.annotations.SerializedName; -import org.apache.commons.lang3.StringUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -47,7 +45,6 @@ import java.util.Collections; import java.util.HashMap; import java.util.List; -import java.util.Locale; import java.util.Map; import java.util.Objects; import java.util.Optional; @@ -79,20 +76,6 @@ */ public class S3Resource extends Resource { private static final Logger LOG = LogManager.getLogger(S3Resource.class); - // Only aliases already supported by GCSProperties belong here. Resource/Vault options - // without a GCS alias (bucket, root path, region, validity checking) keep their existing names. - private static final Map GCS_PROPERTY_ALIASES = ImmutableMap.builder() - .put("gs.endpoint", S3ResourceCompat.ENDPOINT) - .put("gs.access_key", S3ResourceCompat.ACCESS_KEY) - .put("gs.secret_key", S3ResourceCompat.SECRET_KEY) - .put("gs.session_token", S3ResourceCompat.SESSION_TOKEN) - .put("gs.connection.maximum", S3ResourceCompat.MAX_CONNECTIONS) - .put("gs.connection.request.timeout", S3ResourceCompat.REQUEST_TIMEOUT_MS) - .put("gs.connection.timeout", S3ResourceCompat.CONNECTION_TIMEOUT_MS) - .put("gs.use_path_style", S3ResourceCompat.USE_PATH_STYLE) - .put("gs.force_parsing_by_standard_uri", "force_parsing_by_standard_uri") - .build(); - @SerializedName(value = "properties") private Map properties; @@ -112,8 +95,7 @@ public String getProperty(String propertyKey) { @Override protected void setProperties(ImmutableMap newProperties) throws DdlException { Preconditions.checkState(newProperties != null); - this.properties = Maps.newHashMap(newProperties); - normalizeProperties(this.properties, this.properties.get("provider")); + this.properties = StorageAdapter.normalizeProperties(newProperties, newProperties); // check properties S3ResourceCompat.requiredS3PingProperties(properties); @@ -238,62 +220,29 @@ protected static void pingS3(String bucketName, String rootPath, Map properties, String provider) { - // Validate the raw aliases before a preferred value can hide conflicting credentials. - GcsAuthResolver.resolve(properties); - if (StringUtils.isBlank(provider) && GcsAuthResolver.guessIsGcs(properties)) { - provider = "GCP"; - properties.put("provider", provider); - } - if (StringUtils.isBlank(provider)) { - return; - } - switch (provider.toUpperCase(Locale.ROOT)) { - case "GCP": - // Normalize before validation, policy checks and persistence, so FE connector - // binding and Resource/Vault protocol builders consume the same values. - GCS_PROPERTY_ALIASES.forEach((alias, key) -> { - String value = properties.remove(alias); - // Match connector binding: nonblank gs.* values take precedence. - if (StringUtils.isNotBlank(value)) { - properties.put(key, value); - } - }); - break; - default: - break; - } - } - @Override public synchronized void modifyProperties(Map newProperties) throws DdlException { // Serialize the snapshot, validation and publication. A lock only around publication // would allow a concurrent ALTER to replace a successful update with an older snapshot. Map properties = new HashMap<>(newProperties); - String provider = StringUtils.defaultIfEmpty(properties.get("provider"), - this.properties.get("provider")); - if (StringUtils.isBlank(provider)) { - Map selectionProperties = new HashMap<>(this.properties); - selectionProperties.putAll(properties); - if (GcsAuthResolver.guessIsGcs(selectionProperties)) { - provider = "GCP"; - properties.put("provider", provider); - } + Map selectionProperties = new HashMap<>(this.properties); + selectionProperties.putAll(properties); + if (Strings.isNullOrEmpty(properties.get("provider"))) { + // Empty ALTER values retain the persisted provider, like other resource properties. + selectionProperties.put("provider", this.properties.get("provider")); } - // Preserve AWS_* ALTER compatibility before merging with stored canonical properties. + // Normalize the patch independently so its aliases win over persisted canonical values. S3ResourceCompat.convertToStdProperties(properties); - // Resolve aliases separately so this ALTER wins over persisted values regardless - // of their spelling. Within each map, nonblank gs.* values still take precedence. - normalizeProperties(properties, provider); - Map effectiveProperties = new HashMap<>(this.properties); - normalizeProperties(effectiveProperties, provider); + Map normalizedUpdates = StorageAdapter.normalizeProperties(properties, selectionProperties); + Map effectiveProperties = + StorageAdapter.normalizeProperties(this.properties, selectionProperties); S3ResourceCompat.convertToStdProperties(effectiveProperties); - for (Map.Entry update : properties.entrySet()) { + for (Map.Entry update : normalizedUpdates.entrySet()) { // Empty updates are ignored, except when clearing a session token or impersonation account. replaceIfEffectiveValue(effectiveProperties, update.getKey(), update.getValue()); if (S3ResourceCompat.SESSION_TOKEN.equals(update.getKey()) || S3ResourceCompat.Env.TOKEN.equals(update.getKey()) - || GcpCredential.IMPERSONATION_SERVICE_ACCOUNT.equals(update.getKey())) { + || StorageAdapter.isClearableProperty(update.getKey())) { effectiveProperties.put(update.getKey(), update.getValue()); } } @@ -304,7 +253,7 @@ public synchronized void modifyProperties(Map newProperties) thr S3ResourceCompat.Env.REGION, S3ResourceCompat.Env.ROOT_PATH, S3ResourceCompat.Env.BUCKET); Optional any = cantChangeProperties.stream() - .filter(key -> properties.containsKey(key) + .filter(key -> normalizedUpdates.containsKey(key) || !Objects.equals(this.properties.get(key), effectiveProperties.get(key))) .findAny(); if (any.isPresent()) { @@ -314,7 +263,7 @@ public synchronized void modifyProperties(Map newProperties) thr if (!Strings.isNullOrEmpty(effectiveProperties.get(S3ResourceCompat.ENDPOINT))) { effectiveProperties.put(S3ResourceCompat.Env.ENDPOINT, effectiveProperties.get(S3ResourceCompat.ENDPOINT)); } - for (Map.Entry kv : properties.entrySet()) { + for (Map.Entry kv : normalizedUpdates.entrySet()) { if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ROLE_ARN) && !Strings.isNullOrEmpty(kv.getValue())) { effectiveProperties.remove(S3ResourceCompat.ACCESS_KEY); @@ -330,7 +279,7 @@ public synchronized void modifyProperties(Map newProperties) thr effectiveProperties.remove(S3ResourceCompat.Env.EXTERNAL_ID); } } - GcsAuthResolver.resolve(effectiveProperties); + StorageAdapter.resolveAuthentication(effectiveProperties); boolean needCheck = isNeedCheck(effectiveProperties); if (LOG.isDebugEnabled()) { LOG.debug("s3 info need check validity : {}", needCheck); diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java index 60ae2c88b56155..1546a2b687f23e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3StorageVault.java @@ -71,8 +71,6 @@ public static class PropertyKey { public static final String ROLE_ARN = "s3.role_arn"; public static final String EXTERNAL_ID = "s3.external_id"; public static final String CREDENTIALS_PROVIDER_TYPE = "s3.credentials_provider_type"; - public static final String GCP_CREDENTIAL_PROVIDER_TYPE = "gs.credential_provider_type"; - public static final String GCP_IMPERSONATION_SERVICE_ACCOUNT = "gs.impersonation_service_account"; } public static final HashSet ALLOW_ALTER_PROPERTIES = new HashSet<>(Arrays.asList( @@ -83,9 +81,7 @@ public static class PropertyKey { PropertyKey.USE_PATH_STYLE, PropertyKey.ROLE_ARN, PropertyKey.EXTERNAL_ID, - PropertyKey.CREDENTIALS_PROVIDER_TYPE, - PropertyKey.GCP_CREDENTIAL_PROVIDER_TYPE, - PropertyKey.GCP_IMPERSONATION_SERVICE_ACCOUNT + PropertyKey.CREDENTIALS_PROVIDER_TYPE )); @SerializedName(value = "properties") diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java index 643bdd6f1d1f2f..4f3d1c07c30969 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java @@ -27,8 +27,7 @@ import org.apache.doris.common.lock.MonitoredReentrantReadWriteLock; import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; import org.apache.doris.datasource.storage.S3ResourceCompat; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.proto.InternalService.PAlterVaultSyncRequest; import org.apache.doris.rpc.BackendServiceProxy; @@ -162,6 +161,7 @@ private Cloud.StorageVaultPB.Builder buildAlterS3VaultRequest(Map propert .setRequestIp(FrontendOptions.getLocalHostAddressCached()); if (type == StorageVaultType.S3) { properties.keySet().stream() - .filter(key -> !S3StorageVault.ALLOW_ALTER_PROPERTIES.contains(key)) + .filter(key -> !S3StorageVault.ALLOW_ALTER_PROPERTIES.contains(key) + && !StorageAdapter.isModifiableCredentialProperty(key)) .findAny() .ifPresent(key -> { throw new IllegalArgumentException("Alter property " + key + " is not allowed."); diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java index 3f565af192c397..5b96eebbbd047f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java @@ -22,6 +22,7 @@ import org.apache.doris.cloud.proto.Cloud.StagePB.StageAccessType; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Config; +import org.apache.doris.datasource.property.storage.auth.GcpCredentialAdapter; import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.filesystem.spi.StsCredentials; @@ -79,10 +80,7 @@ public static StorageAdapter toStorageAdapter(ObjectInfo obj) { Map gcpProperties = buildS3CompatibleProps(obj); gcpProperties.put("provider", "GCP"); if (obj.getGcpCredential() != null) { - gcpProperties.put("gs.credential_provider_type", - obj.getGcpCredential().getCredentialProviderType().name()); - putIfNotBlank(gcpProperties, "gs.impersonation_service_account", - obj.getGcpCredential().getImpersonationServiceAccount()); + gcpProperties.putAll(GcpCredentialAdapter.toProperties(obj.getGcpCredential())); } return StorageAdapter.ofProvider("GCS", gcpProperties); case COS: diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java index 073529602235c0..bcdc0f59990d1d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java @@ -19,24 +19,31 @@ import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcpCredentialProviderType; +import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.thrift.TCredential; import org.apache.doris.thrift.TGcpCredential; import org.apache.doris.thrift.TGcpCredentialProviderType; import org.apache.doris.thrift.TObjStorageType; import org.apache.doris.thrift.TS3StorageParam; -import org.apache.commons.lang3.StringUtils; +import java.util.HashMap; +import java.util.Map; /** GCP native credential shared by the Cloud protobuf and BE Thrift paths. */ public final class GcpCredentialAdapter implements ObjCredential { - private final GcpCredentialProviderType providerType; - private final String impersonationServiceAccount; + private final Map fields; - public GcpCredentialAdapter(GcpCredential credential) { - providerType = credential.getCredentialProviderType(); - impersonationServiceAccount = credential.getImpersonationServiceAccount(); + public GcpCredentialAdapter(Map fields) { + this.fields = Map.copyOf(fields); + } + + public static Map toProperties(GcpCredentialPB credential) { + Map fields = new HashMap<>(); + fields.put("credential_provider_type", credential.getCredentialProviderType().name()); + if (credential.hasImpersonationServiceAccount()) { + fields.put("impersonation_service_account", credential.getImpersonationServiceAccount()); + } + return StorageAdapter.credentialToProperties("GCS", fields); } @Override @@ -44,12 +51,13 @@ public void applyTo(ObjectStoreInfoPB.Builder builder) { if (!builder.hasProvider()) { builder.setProvider(ObjectStoreInfoPB.Provider.GCP); } - GcpCredentialPB.Builder credential = GcpCredentialPB.newBuilder() - .setCredentialProviderType(providerType == GcpCredentialProviderType.DEFAULT - ? GcpCredentialPB.CredentialProviderType.DEFAULT - : GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE); - if (StringUtils.isNotBlank(impersonationServiceAccount)) { - credential.setImpersonationServiceAccount(impersonationServiceAccount); + GcpCredentialPB.Builder credential = GcpCredentialPB.newBuilder(); + if (fields.containsKey("credential_provider_type")) { + credential.setCredentialProviderType(GcpCredentialPB.CredentialProviderType.valueOf( + fields.get("credential_provider_type"))); + } + if (fields.containsKey("impersonation_service_account")) { + credential.setImpersonationServiceAccount(fields.get("impersonation_service_account")); } builder.getCredentialBuilder().setGcpCredential(credential); } @@ -59,12 +67,13 @@ public void applyTo(TS3StorageParam param) { if (!param.isSetProvider()) { param.setProvider(TObjStorageType.GCP); } - TGcpCredential credential = new TGcpCredential() - .setCredentialProviderType(providerType == GcpCredentialProviderType.DEFAULT - ? TGcpCredentialProviderType.DEFAULT - : TGcpCredentialProviderType.COMPUTE_ENGINE); - if (StringUtils.isNotBlank(impersonationServiceAccount)) { - credential.setImpersonationServiceAccount(impersonationServiceAccount); + TGcpCredential credential = new TGcpCredential(); + if (fields.containsKey("credential_provider_type")) { + credential.setCredentialProviderType(TGcpCredentialProviderType.valueOf( + fields.get("credential_provider_type"))); + } + if (fields.containsKey("impersonation_service_account")) { + credential.setImpersonationServiceAccount(fields.get("impersonation_service_account")); } param.setCredential(new TCredential().setGcpCredential(credential)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java index 1e8dc0928f876e..82f68aa4566661 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java @@ -17,50 +17,30 @@ package org.apache.doris.datasource.property.storage.auth; -import org.apache.doris.filesystem.auth.GcsAuth; -import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; -import java.util.Arrays; -import java.util.List; import java.util.Map; import java.util.Optional; -/** Selects exactly one provider-native credential from object-storage properties. */ +/** Converts plugin-resolved native credentials to the core wire protocols. */ public final class ObjCredentialFactory { - private interface Parser { - Optional parse(Map properties, Optional gcsAuth); - } - - // Add one parser here when introducing OSS, Azure, or another native credential. - // Provider-specific resolvers select credentials; credential classes handle transport conversion. - private static final List PARSERS = Arrays.asList((properties, gcsAuth) -> - gcsAuth.flatMap(GcsAuth::getNativeCredential).map(GcpCredentialAdapter::new)); - - // TODO: Add AWS role-based authentication after its existing property and - // transport fields can be migrated without breaking compatibility. Static - // AK/SK credentials remain outside this provider-native credential model. - private ObjCredentialFactory() { } public static Optional fromProperties(Map properties) { - return fromProperties(properties, GcsAuthResolver.resolve(properties)); + return fromAuthentication(StorageAdapter.resolveAuthentication(properties)); + } + + public static Optional fromAuthentication(Optional authentication) { + return authentication.filter(ObjectStorageAuthentication::isNative) + .map(auth -> fromCredential(auth.getProvider(), auth.getCredential())); } - /** Reuse the authentication choice already resolved by a protocol builder. */ - public static Optional fromProperties(Map properties, Optional gcsAuth) { - ObjCredential selected = null; - for (Parser parser : PARSERS) { - Optional parsed = parser.parse(properties, gcsAuth); - if (!parsed.isPresent()) { - continue; - } - if (selected != null) { - throw new IllegalArgumentException( - "Only one provider-native object storage credential may be configured."); - } - selected = parsed.get(); + public static ObjCredential fromCredential(String provider, Map credential) { + if ("GCP".equals(provider)) { + return new GcpCredentialAdapter(credential); } - return Optional.ofNullable(selected); + throw new IllegalArgumentException("Unsupported native credential wire format: " + provider); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java index 2e179d0bdf32fd..69166f27a81e13 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java @@ -22,8 +22,7 @@ import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB.Provider; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; import org.apache.doris.datasource.property.storage.auth.ObjCredentialFactory; -import org.apache.doris.filesystem.auth.GcsAuth; -import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import com.google.common.base.Preconditions; @@ -64,8 +63,8 @@ private CloudObjectStoreAdapter() { /** Direct move of legacy {@code S3Properties.getObjStoreInfoPB}. */ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map properties) { - Optional gcsAuth = GcsAuthResolver.resolve(properties); - if (gcsAuth.filter(GcsAuth::isAnonymous).isPresent()) { + Optional authentication = StorageAdapter.resolveAuthentication(properties); + if (authentication.filter(ObjectStorageAuthentication::isAnonymous).isPresent()) { throw new IllegalArgumentException("Anonymous GCS authentication is not supported for storage vaults."); } Cloud.ObjectStoreInfoPB.Builder builder = Cloud.ObjectStoreInfoPB.newBuilder(); @@ -123,11 +122,20 @@ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map credential.applyTo(builder)); return builder; } + /** ALTER carries only explicitly supplied credential fields, including empty values. */ + public static void applyCredentialPatch(Cloud.ObjectStoreInfoPB.Builder builder, Map properties) { + builder.clearCredential(); + StorageAdapter.resolveAuthentication(properties) + .filter(auth -> !auth.getCredentialUpdates().isEmpty()) + .ifPresent(auth -> ObjCredentialFactory.fromCredential(auth.getProvider(), auth.getCredentialUpdates()) + .applyTo(builder)); + } + private static boolean hasCredentialsProviderType(Map properties) { return properties.containsKey(CREDENTIALS_PROVIDER_TYPE) || properties.containsKey(ENV_CREDENTIALS_PROVIDER_TYPE); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java index 9dfc255b2b0066..48f26359194792 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java @@ -136,7 +136,7 @@ public static void optionalS3Property(Map properties) { } public static void convertToStdProperties(Map properties) { - org.apache.doris.filesystem.auth.GcsAuthResolver.resolve(properties); + StorageAdapter.resolveAuthentication(properties); if (properties.containsKey(Env.ENDPOINT)) { properties.putIfAbsent(ENDPOINT, properties.get(Env.ENDPOINT)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java index 37ff1baed4cd67..843b3b5e92c152 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java @@ -19,8 +19,7 @@ import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; import org.apache.doris.datasource.property.storage.auth.ObjCredentialFactory; -import org.apache.doris.filesystem.auth.GcsAuth; -import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.thrift.TCredProviderType; import org.apache.doris.thrift.TObjStorageType; @@ -72,7 +71,7 @@ private S3ThriftAdapter() { /** Direct move of legacy {@code S3Properties.getS3TStorageParam}. */ public static TS3StorageParam getS3TStorageParam(Map properties) { - Optional gcsAuth = GcsAuthResolver.resolve(properties); + Optional authentication = StorageAdapter.resolveAuthentication(properties); TS3StorageParam s3Info = new TS3StorageParam(); if (StringUtils.isNotBlank(properties.get("provider"))) { String provider = properties.get("provider").trim().toUpperCase(Locale.ROOT); @@ -107,10 +106,10 @@ public static TS3StorageParam getS3TStorageParam(Map properties) ? DEFAULT_CONNECTION_TIMEOUT_MS : connTimeoutMs)); String usePathStyle = properties.getOrDefault(USE_PATH_STYLE, "false"); s3Info.setUsePathStyle(Boolean.parseBoolean(usePathStyle)); - ObjCredentialFactory.fromProperties(properties, gcsAuth) + ObjCredentialFactory.fromAuthentication(authentication) .ifPresent(credential -> credential.applyTo(s3Info)); - if (gcsAuth.filter(GcsAuth::isAnonymous).isPresent() - || (gcsAuth.isEmpty() && getCredentialsProviderMode(properties, + if (authentication.filter(ObjectStorageAuthentication::isAnonymous).isPresent() + || (authentication.isEmpty() && getCredentialsProviderMode(properties, AwsCredentialsProviderMode.INSTANCE_PROFILE) == AwsCredentialsProviderMode.ANONYMOUS)) { s3Info.setCredProviderType(TCredProviderType.ANONYMOUS); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index 1648a188458379..4e726f2780f328 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -19,6 +19,7 @@ import org.apache.doris.common.Config; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.FileSystemProperties; import org.apache.doris.filesystem.properties.FsCacheKeys; import org.apache.doris.filesystem.properties.HadoopStorageProperties; @@ -39,6 +40,7 @@ import java.util.Locale; import java.util.Map; import java.util.Objects; +import java.util.Optional; import java.util.Set; /** @@ -141,6 +143,48 @@ private StorageAdapter(FileSystemProperties spi, Map origProps, checkAzureOauth2OnlyForIcebergRest(); } + /** Provider-owned normalization also accepts partial resource/vault ALTER maps. */ + public static Map normalizeProperties(Map properties, + Map context) { + Map normalized = new HashMap<>(properties); + for (FileSystemProvider provider : manager().getProviders()) { + normalized = provider.normalizeProperties(normalized, context); + } + return normalized; + } + + public static Optional resolveAuthentication(Map properties) { + ObjectStorageAuthentication selected = null; + for (FileSystemProvider provider : manager().getProviders()) { + Optional auth = provider.resolveAuthentication(properties); + if (auth.isPresent()) { + if (selected != null) { + throw new IllegalArgumentException("Multiple filesystem providers claimed object authentication"); + } + selected = auth.get(); + } + } + return Optional.ofNullable(selected); + } + + public static boolean isModifiableCredentialProperty(String key) { + return manager().getProviders().stream() + .anyMatch(provider -> provider.modifiableCredentialPropertyKeys().contains(key)); + } + + public static boolean isClearableProperty(String key) { + return manager().getProviders().stream().anyMatch(provider -> provider.clearablePropertyKeys().contains(key)); + } + + public static Map credentialToProperties(String providerName, Map credential) { + for (FileSystemProvider provider : manager().getProviders()) { + if (provider.name().equalsIgnoreCase(providerName)) { + return provider.credentialToProperties(credential); + } + } + throw new StoragePropertiesException("Filesystem provider '" + providerName + "' is not available"); + } + /** * Binds the primary provider for the raw user properties, mirroring the legacy * {@code StorageProperties.createPrimary}. @@ -505,7 +549,7 @@ private Map alignS3FamilyBackendMap(S3CompatibleFileSystemProper aligned.remove("AWS_ROLE_ARN"); aligned.remove("AWS_EXTERNAL_ID"); if (s3.hasStaticCredentials() - || aligned.containsKey(org.apache.doris.filesystem.auth.GcpCredential.CREDENTIAL_PROVIDER_TYPE)) { + || resolveAuthentication(aligned).map(ObjectStorageAuthentication::isNative).orElse(false)) { aligned.remove("AWS_CREDENTIALS_PROVIDER_TYPE"); } else { aligned.put("AWS_CREDENTIALS_PROVIDER_TYPE", AwsCredentialsProviderMode.ANONYMOUS.name()); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java index 25cb81a5d62fbf..68072e070a61fd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java @@ -28,7 +28,7 @@ import org.apache.doris.common.Config; import org.apache.doris.common.FeConstants; import org.apache.doris.common.util.DatasourcePrintableMap; -import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.datasource.property.storage.auth.GcpCredentialAdapter; import org.apache.doris.mysql.privilege.AccessControllerManager; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.PlanType; @@ -132,12 +132,7 @@ private String getObjectCreateStmt(Cloud.ObjectStoreInfoPB objectInfo) { properties.put("s3.root.path", objectInfo.getPrefix()); properties.put("s3.bucket", objectInfo.getBucket()); if (objectInfo.hasCredential() && objectInfo.getCredential().hasGcpCredential()) { - Cloud.GcpCredentialPB credential = objectInfo.getCredential().getGcpCredential(); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); - if (!credential.getImpersonationServiceAccount().isEmpty()) { - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, - credential.getImpersonationServiceAccount()); - } + properties.putAll(GcpCredentialAdapter.toProperties(objectInfo.getCredential().getGcpCredential())); } else { // Masking even empty keys would produce nonempty HMAC credentials conflicting with native auth. properties.put("s3.access_key", objectInfo.getAk()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java index 24326f9ae00fd9..8b9ad26cdb0b32 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/StorageDescPersistTest.java @@ -18,7 +18,6 @@ package org.apache.doris.analysis; import org.apache.doris.datasource.storage.StorageAdapter; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.load.EtlJobType; import org.apache.doris.load.loadv2.BrokerLoadJob; import org.apache.doris.persist.gson.GsonUtils; @@ -82,7 +81,7 @@ public void testGcpNativeCredentialForExportBroker() throws Exception { Map properties = Maps.newHashMap(); properties.put("provider", "GCP"); properties.put("s3.region", "us-east1"); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + properties.put("gs.credential_provider_type", "DEFAULT"); // EXPORT forwards this BrokerDesc map to BE Writer. Verify the shared FE conversion // preserves the provider-specific credential while keeping the S3-compatible file type. @@ -91,7 +90,7 @@ public void testGcpNativeCredentialForExportBroker() throws Exception { Assertions.assertEquals(TFileType.FILE_S3, brokerDesc.getFileType()); Assertions.assertEquals("GCP", brokerDesc.getBackendConfigProperties().get("provider")); Assertions.assertEquals("DEFAULT", brokerDesc.getBackendConfigProperties() - .get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + .get("gs.credential_provider_type")); Assertions.assertEquals("s3://export-bucket/path", brokerDesc.getFileLocation("gs://export-bucket/path")); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 57182bcdd37819..0a4174326ee985 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -25,7 +25,6 @@ import org.apache.doris.datasource.storage.S3ResourceCompat; import org.apache.doris.datasource.storage.S3ThriftAdapter; import org.apache.doris.datasource.storage.StorageAdapter; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.meta.MetaContext; import org.apache.doris.mysql.privilege.AccessControllerManager; @@ -748,12 +747,12 @@ public void testAlterClearsGcpImpersonation() throws Exception { S3Resource resource = createGcpResourceWithImpersonation(providerType); resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.CONNECTION_TIMEOUT_MS, "2000")); Assertions.assertEquals("reader@project.iam.gserviceaccount.com", - resource.getProperty(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + resource.getProperty("gs.impersonation_service_account")); AtomicBoolean pinged = new AtomicBoolean(); resourceMock.when(() -> S3Resource.pingS3(Mockito.anyString(), Mockito.anyString(), Mockito.anyMap())) .thenAnswer(invocation -> { Map pingProperties = invocation.getArgument(2); - Assertions.assertEquals("", pingProperties.get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + Assertions.assertEquals("", pingProperties.get("gs.impersonation_service_account")); TGcpCredential credential = S3ThriftAdapter.getS3TStorageParam(pingProperties) .getCredential().getGcpCredential(); Assertions.assertEquals(providerType, credential.getCredentialProviderType().name()); @@ -762,11 +761,11 @@ public void testAlterClearsGcpImpersonation() throws Exception { return null; }); long previousVersion = resource.getVersion(); - resource.modifyProperties(ImmutableMap.of(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "", + resource.modifyProperties(ImmutableMap.of("gs.impersonation_service_account", "", S3ResourceCompat.VALIDITY_CHECK, "true")); Assertions.assertTrue(pinged.get()); Assertions.assertEquals(previousVersion + 1, resource.getVersion()); - Assertions.assertEquals("", resource.getProperty(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + Assertions.assertEquals("", resource.getProperty("gs.impersonation_service_account")); ByteArrayOutputStream bytes = new ByteArrayOutputStream(); try (DataOutputStream output = new DataOutputStream(bytes)) { @@ -797,7 +796,7 @@ public void testFailedAlterPreservesGcpImpersonation() throws Exception { resourceMock.when(() -> S3Resource.pingS3(Mockito.anyString(), Mockito.anyString(), Mockito.anyMap())) .thenAnswer(invocation -> { Map pingProperties = invocation.getArgument(2); - Assertions.assertEquals("", pingProperties.get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + Assertions.assertEquals("", pingProperties.get("gs.impersonation_service_account")); throw new DdlException("source credential cannot access bucket"); }); for (String providerType : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { @@ -805,7 +804,7 @@ public void testFailedAlterPreservesGcpImpersonation() throws Exception { Map previousProperties = resource.getCopiedProperties(); long previousVersion = resource.getVersion(); Assertions.assertThrows(DdlException.class, () -> resource.modifyProperties( - ImmutableMap.of(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "", + ImmutableMap.of("gs.impersonation_service_account", "", S3ResourceCompat.VALIDITY_CHECK, "true"))); Assertions.assertEquals(previousProperties, resource.getCopiedProperties()); Assertions.assertEquals(previousVersion, resource.getVersion()); @@ -821,8 +820,8 @@ private S3Resource createGcpResourceWithImpersonation(String providerType) throw properties.put(S3ResourceCompat.BUCKET, "bucket"); properties.put(S3ResourceCompat.ROOT_PATH, "prefix"); properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, providerType); - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "reader@project.iam.gserviceaccount.com"); + properties.put("gs.credential_provider_type", providerType); + properties.put("gs.impersonation_service_account", "reader@project.iam.gserviceaccount.com"); S3Resource resource = new S3Resource("gcp_resource"); resource.setProperties(ImmutableMap.copyOf(properties)); return resource; @@ -835,7 +834,7 @@ public void testGsCredentialAliasesCannotBypassNativeAuthValidation() throws Exc properties.put("gs.endpoint", "https://storage.googleapis.com"); properties.put(S3ResourceCompat.BUCKET, "bucket"); properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + properties.put("gs.credential_provider_type", "DEFAULT"); for (String key : new String[] {"gs.access_key", "gs.secret_key", "gs.session_token", S3ResourceCompat.ACCESS_KEY, S3ResourceCompat.SECRET_KEY, S3ResourceCompat.SESSION_TOKEN}) { S3Resource resource = new S3Resource("gcp_resource"); @@ -849,18 +848,18 @@ public void testGsCredentialAliasesCannotBypassNativeAuthValidation() throws Exc Assertions.assertThrows(IllegalArgumentException.class, () -> resource.modifyProperties(ImmutableMap.of(key, "conflicting-credential"))); Assertions.assertEquals(before, resource.getCopiedProperties()); - Assertions.assertEquals("DEFAULT", resource.getProperty(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals("DEFAULT", resource.getProperty("gs.credential_provider_type")); } S3Resource resource = new S3Resource("gcp_resource"); - properties.remove(GcpCredential.CREDENTIAL_PROVIDER_TYPE); + properties.remove("gs.credential_provider_type"); properties.put("gs.access_key", "access"); properties.put("gs.secret_key", "secret"); resource.setProperties(ImmutableMap.copyOf(properties)); Assertions.assertThrows(IllegalArgumentException.class, () -> resource.modifyProperties( - ImmutableMap.of(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"))); + ImmutableMap.of("gs.credential_provider_type", "DEFAULT"))); // Empty AK/SK updates are ignored by Resource ALTER and cannot clear stored credentials. Assertions.assertThrows(IllegalArgumentException.class, () -> resource.modifyProperties( - ImmutableMap.of(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT", + ImmutableMap.of("gs.credential_provider_type", "DEFAULT", S3ResourceCompat.ACCESS_KEY, "", S3ResourceCompat.SECRET_KEY, ""))); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java index 619eab10f80080..a407a9c848fba0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java @@ -25,7 +25,6 @@ import org.apache.doris.cloud.rpc.MetaServiceProxy; import org.apache.doris.common.DdlException; import org.apache.doris.datasource.storage.S3ResourceCompat; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.system.SystemInfoService; @@ -130,13 +129,13 @@ public void testAwsAlterPreservesOmissionAndExplicitClears() throws Exception { @Test public void testGcpCreateDefaultsAndExplicitProvider() throws Exception { Map properties = new HashMap<>(); - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ACCOUNT); + properties.put("gs.impersonation_service_account", ACCOUNT); Cloud.GcpCredentialPB credential = captureRequest(properties, false).getCredential().getGcpCredential(); Assertions.assertTrue(credential.hasCredentialProviderType()); Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.DEFAULT, credential.getCredentialProviderType()); Assertions.assertEquals(ACCOUNT, credential.getImpersonationServiceAccount()); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "COMPUTE_ENGINE"); + properties.put("gs.credential_provider_type", "COMPUTE_ENGINE"); credential = captureRequest(properties, false).getCredential().getGcpCredential(); Assertions.assertTrue(credential.hasCredentialProviderType()); Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE, @@ -146,7 +145,7 @@ public void testGcpCreateDefaultsAndExplicitProvider() throws Exception { @Test public void testGcpEmptyTargetDefaultsToDefault() throws Exception { Map properties = new HashMap<>(); - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ""); + properties.put("gs.impersonation_service_account", ""); Cloud.GcpCredentialPB credential = captureRequest(properties, false).getCredential().getGcpCredential(); Assertions.assertTrue(credential.hasCredentialProviderType()); Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.DEFAULT, @@ -162,19 +161,19 @@ public void testGcpEmptyTargetDefaultsToDefault() throws Exception { @Test public void testGcpAlterPreservesOmittedFields() throws Exception { Map properties = new HashMap<>(); - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ACCOUNT); + properties.put("gs.impersonation_service_account", ACCOUNT); Cloud.GcpCredentialPB credential = captureRequest(properties, true).getCredential().getGcpCredential(); Assertions.assertFalse(credential.hasCredentialProviderType()); Assertions.assertTrue(credential.hasImpersonationServiceAccount()); Assertions.assertEquals(ACCOUNT, credential.getImpersonationServiceAccount()); properties.clear(); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "COMPUTE_ENGINE"); + properties.put("gs.credential_provider_type", "COMPUTE_ENGINE"); credential = captureRequest(properties, true).getCredential().getGcpCredential(); Assertions.assertTrue(credential.hasCredentialProviderType()); Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE, credential.getCredentialProviderType()); Assertions.assertFalse(credential.hasImpersonationServiceAccount()); - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, ACCOUNT); + properties.put("gs.impersonation_service_account", ACCOUNT); credential = captureRequest(properties, true).getCredential().getGcpCredential(); Assertions.assertTrue(credential.hasCredentialProviderType()); Assertions.assertEquals(Cloud.GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE, @@ -208,7 +207,7 @@ public void testGcpAnonymousAliasesAreRejectedBeforeCreateRpc() throws Exception mockedProxy.when(MetaServiceProxy::getInstance).thenReturn(proxy); mockedEnv.when(Env::getCurrentEnv).thenReturn(env); StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); - for (String key : new String[] {GcpCredential.CREDENTIAL_PROVIDER_TYPE, + for (String key : new String[] {"gs.credential_provider_type", S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE}) { Map properties = new HashMap<>(); properties.put("type", "S3"); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java index aadb59dfbb975b..5016cd0c0c6dff 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/OutFileTest.java @@ -17,7 +17,6 @@ package org.apache.doris.nereids.trees.plans; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator; @@ -110,10 +109,9 @@ public void testGcsOutFileNormalizesPathAndPreservesCredentials() throws Excepti String properties = "\"provider\" = \"GCP\"," + "\"s3.endpoint\" = \"https://storage.googleapis.com\"," + "\"s3.region\" = \"us-central1\"," - + "\"" + GcpCredential.CREDENTIAL_PROVIDER_TYPE + "\" = \"" + provider + "\""; + + "\"gs.credential_provider_type\" = \"" + provider + "\""; if (!serviceAccount.isEmpty()) { - properties += ",\"" + GcpCredential.IMPERSONATION_SERVICE_ACCOUNT - + "\" = \"" + serviceAccount + "\""; + properties += ",\"gs.impersonation_service_account\" = \"" + serviceAccount + "\""; } TResultFileSinkOptions options = getFileSinkOptions("select * from T1 into outfile '" + scheme + "://outfile-bucket/nested/result_' format as csv properties (" @@ -124,9 +122,9 @@ public void testGcsOutFileNormalizesPathAndPreservesCredentials() throws Excepti Assertions.assertEquals("https://storage.googleapis.com", backendProperties.get("AWS_ENDPOINT")); // GCSProperties does not bind s3.region and retains its compatibility default. Assertions.assertEquals("us-east1", backendProperties.get("AWS_REGION")); - Assertions.assertEquals(provider, backendProperties.get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals(provider, backendProperties.get("gs.credential_provider_type")); Assertions.assertEquals(serviceAccount, - backendProperties.getOrDefault(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "")); + backendProperties.getOrDefault("gs.impersonation_service_account", "")); } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java index 53d24e4c935afd..4007ee6634db43 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java @@ -22,7 +22,6 @@ import org.apache.doris.catalog.info.TableNameInfo; import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.load.ExportJob; import org.apache.doris.qe.ConnectContext; import org.apache.doris.utframe.TestWithFeService; @@ -81,9 +80,9 @@ public void testGcsExportNormalizesPathAndPreservesCredentials() throws Exceptio Map properties = new HashMap<>(); properties.put("provider", "GCP"); properties.put("s3.endpoint", "https://storage.googleapis.com"); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, provider); + properties.put("gs.credential_provider_type", provider); if (!serviceAccount.isEmpty()) { - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, serviceAccount); + properties.put("gs.impersonation_service_account", serviceAccount); } ExportJob job = generateExportJob(scheme + "://export-bucket/nested/result_", new BrokerDesc(null, properties)); @@ -91,9 +90,9 @@ public void testGcsExportNormalizesPathAndPreservesCredentials() throws Exceptio Map backendProperties = job.getBrokerDesc().getBackendConfigProperties(); Assertions.assertEquals("GCP", backendProperties.get("provider")); Assertions.assertEquals("https://storage.googleapis.com", backendProperties.get("AWS_ENDPOINT")); - Assertions.assertEquals(provider, backendProperties.get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertEquals(provider, backendProperties.get("gs.credential_provider_type")); Assertions.assertEquals(serviceAccount, - backendProperties.getOrDefault(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "")); + backendProperties.getOrDefault("gs.impersonation_service_account", "")); } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java index c101eb6f8aae08..a15130636d8f03 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/LoadCommandTest.java @@ -22,7 +22,6 @@ import org.apache.doris.common.Pair; import org.apache.doris.datasource.property.fileformat.CsvFileFormatProperties; import org.apache.doris.datasource.property.fileformat.DeferredFileFormatProperties; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.analyzer.UnboundSlot; import org.apache.doris.nereids.load.NereidsDataDescription; @@ -103,9 +102,9 @@ public void testGcpNativeCredentialForS3Load() { Assertions.assertEquals("us-east1", brokerDesc.getBackendConfigProperties().get("AWS_REGION")); Assertions.assertEquals("GCP", brokerDesc.getBackendConfigProperties().get("provider")); Assertions.assertEquals("COMPUTE_ENGINE", brokerDesc.getBackendConfigProperties() - .get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + .get("gs.credential_provider_type")); Assertions.assertEquals("target@my-project.iam.gserviceaccount.com", - brokerDesc.getBackendConfigProperties().get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + brokerDesc.getBackendConfigProperties().get("gs.impersonation_service_account")); } @Test @@ -120,7 +119,7 @@ public void testGcpNativeCredentialValidationHappensBeforeLoadSubmission() { IllegalArgumentException exception = Assertions.assertThrows(IllegalArgumentException.class, () -> new NereidsParser().parseSingle(loadSql)); - Assertions.assertTrue(exception.getMessage().contains(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + Assertions.assertTrue(exception.getMessage().contains("gs.credential_provider_type")); Assertions.assertTrue(exception.getMessage().contains("DEFAULT")); Assertions.assertTrue(exception.getMessage().contains("COMPUTE_ENGINE")); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java index 450b3094b436c8..eb2e2472638c52 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommandTest.java @@ -21,7 +21,6 @@ import org.apache.doris.common.jmockit.Deencapsulation; import org.apache.doris.common.util.DatasourcePrintableMap; import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.nereids.parser.NereidsParser; import org.junit.jupiter.api.Assertions; @@ -46,8 +45,8 @@ public void testNativeGcpIdentityRoundTrip() { original.getCredentialBuilder().setGcpCredential(credential); Map properties = shownProperties(original.build()); - Assertions.assertEquals(providerType.name(), properties.get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); - Assertions.assertEquals(account, properties.getOrDefault(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "")); + Assertions.assertEquals(providerType.name(), properties.get("gs.credential_provider_type")); + Assertions.assertEquals(account, properties.getOrDefault("gs.impersonation_service_account", "")); Assertions.assertFalse(properties.containsKey("s3.access_key")); Assertions.assertFalse(properties.containsKey("s3.secret_key")); Assertions.assertEquals(original.build(), CloudObjectStoreAdapter.getObjStoreInfoPB(properties).build(), @@ -64,8 +63,8 @@ public void testLegacyKeysRemainMasked() { .setAk("access-key").setSk("secret-key").build()); Assertions.assertEquals(DatasourcePrintableMap.PASSWORD_MASK, properties.get("s3.access_key")); Assertions.assertEquals(DatasourcePrintableMap.PASSWORD_MASK, properties.get("s3.secret_key")); - Assertions.assertFalse(properties.containsKey(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); - Assertions.assertFalse(properties.containsKey(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + Assertions.assertFalse(properties.containsKey("gs.credential_provider_type")); + Assertions.assertFalse(properties.containsKey("gs.impersonation_service_account")); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java index 07833c9dec8704..c6adee847b2404 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java @@ -28,7 +28,6 @@ import org.apache.doris.common.Pair; import org.apache.doris.common.util.FileFormatConstants; import org.apache.doris.common.util.FileFormatUtils; -import org.apache.doris.filesystem.auth.GcpCredential; import org.apache.doris.proto.Types.PScalarType; import org.apache.doris.proto.Types.PStructField; import org.apache.doris.proto.Types.PTypeNode; @@ -117,8 +116,8 @@ public void testGcpNativeCredentialForS3Tvf() throws AnalysisException { properties.put("format", "parquet"); properties.put("provider", "GCP"); properties.put("gs.endpoint", "https://storage.googleapis.com"); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "COMPUTE_ENGINE"); - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, + properties.put("gs.credential_provider_type", "COMPUTE_ENGINE"); + properties.put("gs.impersonation_service_account", "target@my-project.iam.gserviceaccount.com"); S3TableValuedFunction tvf = new S3TableValuedFunction(properties); @@ -126,9 +125,9 @@ public void testGcpNativeCredentialForS3Tvf() throws AnalysisException { Assertions.assertEquals("us-east1", tvf.getBackendConnectProperties().get("AWS_REGION")); Assertions.assertEquals("GCP", tvf.getBackendConnectProperties().get("provider")); Assertions.assertEquals("COMPUTE_ENGINE", tvf.getBackendConnectProperties() - .get(GcpCredential.CREDENTIAL_PROVIDER_TYPE)); + .get("gs.credential_provider_type")); Assertions.assertEquals("target@my-project.iam.gserviceaccount.com", - tvf.getBackendConnectProperties().get(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)); + tvf.getBackendConnectProperties().get("gs.impersonation_service_account")); } finally { FeConstants.runningUnitTest = previousRunningUnitTest; } @@ -143,7 +142,7 @@ public void testGcpNativeCredentialValidationHappensBeforeTvfAccess() { properties.put("uri", "gs://tvf-bucket/path/file.parquet"); properties.put("format", "parquet"); properties.put("provider", "GCP"); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + properties.put("gs.credential_provider_type", "DEFAULT"); properties.put("s3.session_token", "session-token"); IllegalArgumentException exception = Assertions.assertThrows(IllegalArgumentException.class, diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java new file mode 100644 index 00000000000000..27e9251acf1980 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java @@ -0,0 +1,63 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import java.util.Map; + +/** + * Provider-resolved authentication, with no property parsing or SDK dependencies. + * + *

The provider identifies the wire credential variant (for example, GCP). Credential map keys + * are fields of that variant, not user-facing configuration keys. Only protocol adapters interpret + * these fields; callers use the authentication mode and pass the maps through unchanged.

+ */ +public final class ObjectStorageAuthentication { + private final String provider; + private final boolean anonymous; + private final Map credential; + private final Map credentialUpdates; + + public ObjectStorageAuthentication(String provider, boolean anonymous, Map credential, + Map credentialUpdates) { + this.provider = provider; + this.anonymous = anonymous; + this.credential = Map.copyOf(credential); + this.credentialUpdates = Map.copyOf(credentialUpdates); + } + + public String getProvider() { + return provider; + } + + public boolean isAnonymous() { + return anonymous; + } + + public boolean isNative() { + return !credential.isEmpty(); + } + + public Map getCredential() { + return credential; + } + + /** Only explicitly supplied native fields, including empty values that clear persisted fields. */ + public Map getCredentialUpdates() { + return credentialUpdates; + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml b/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml new file mode 100644 index 00000000000000..7dfeaa7e60ab22 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml @@ -0,0 +1,42 @@ + + + + 4.0.0 + + + org.apache.doris + fe-filesystem + ${revision} + ../pom.xml + + + fe-filesystem-gcs-common + Doris GCS authentication and property rules + Provider implementation shared by GCS and S3-compatible plugins; not part of the host API. + + + org.junit.jupiter + junit-jupiter + test + + + diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java similarity index 98% rename from fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java rename to fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java index 873ccd5f5b2faa..14130bbc0ecf61 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredential.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.filesystem.auth; +package org.apache.doris.filesystem.gcs.auth; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredentialProviderType.java similarity index 96% rename from fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java rename to fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredentialProviderType.java index 21fe7a85fa6a0b..f8bc0c781fadf0 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialProviderType.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredentialProviderType.java @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.filesystem.auth; +package org.apache.doris.filesystem.gcs.auth; import java.util.Locale; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuth.java similarity index 97% rename from fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java rename to fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuth.java index ee233eb5b0d74a..2d45f2d4301807 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuth.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuth.java @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.filesystem.auth; +package org.apache.doris.filesystem.gcs.auth; import java.util.Optional; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java similarity index 99% rename from fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java rename to fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java index a61542d3499c56..d873f786717fac 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsAuthResolver.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.filesystem.auth; +package org.apache.doris.filesystem.gcs.auth; import java.util.Locale; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsEndpoint.java similarity index 98% rename from fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java rename to fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsEndpoint.java index 9eb84772f2789f..5395a459f82cd5 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcsEndpoint.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsEndpoint.java @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.filesystem.auth; +package org.apache.doris.filesystem.gcs.auth; import java.net.URI; import java.util.regex.Pattern; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java similarity index 99% rename from fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java rename to fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java index b2363a362a2332..c07b730b00c576 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/GcsAuthResolverTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.filesystem.auth; +package org.apache.doris.filesystem.gcs.auth; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java index 87e6c776a816c7..3f3f1c4492e06b 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcpAuth.java @@ -17,8 +17,8 @@ package org.apache.doris.filesystem.gcs; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcsEndpoint; +import org.apache.doris.filesystem.gcs.auth.GcpCredential; +import org.apache.doris.filesystem.gcs.auth.GcsEndpoint; import com.google.auth.oauth2.ComputeEngineCredentials; import com.google.auth.oauth2.GoogleCredentials; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index 90a715749dbe68..74df1420679eb8 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -17,10 +17,10 @@ package org.apache.doris.filesystem.gcs; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcpCredentialProviderType; -import org.apache.doris.filesystem.auth.GcsAuth; -import org.apache.doris.filesystem.auth.GcsAuthResolver; +import org.apache.doris.filesystem.gcs.auth.GcpCredential; +import org.apache.doris.filesystem.gcs.auth.GcpCredentialProviderType; +import org.apache.doris.filesystem.gcs.auth.GcsAuth; +import org.apache.doris.filesystem.gcs.auth.GcsAuthResolver; import org.apache.doris.filesystem.properties.FsCacheKeys; import org.apache.doris.filesystem.s3.AbstractDelegatingS3Properties; import org.apache.doris.foundation.property.ConnectorProperty; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index 716f9a560b2dfe..13aeca870a5560 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -18,7 +18,9 @@ package org.apache.doris.filesystem.gcs; import org.apache.doris.filesystem.FileSystem; -import org.apache.doris.filesystem.auth.GcpCredential; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; +import org.apache.doris.filesystem.gcs.auth.GcpCredential; +import org.apache.doris.filesystem.gcs.auth.GcsAuthResolver; import org.apache.doris.filesystem.s3.S3CompatSignals; import org.apache.doris.filesystem.s3.S3FileSystem; import org.apache.doris.filesystem.s3.S3FileSystemProperties; @@ -28,6 +30,7 @@ import java.io.IOException; import java.util.HashMap; import java.util.Map; +import java.util.Optional; import java.util.Set; /** @@ -45,6 +48,82 @@ public class GcsFileSystemProvider implements FileSystemProvider RESOURCE_ALIASES = Map.ofEntries( + Map.entry("gs.endpoint", "s3.endpoint"), + Map.entry("gs.access_key", "s3.access_key"), + Map.entry("gs.secret_key", "s3.secret_key"), + Map.entry("gs.session_token", "s3.session_token"), + Map.entry("gs.connection.maximum", "s3.connection.maximum"), + Map.entry("gs.connection.request.timeout", "s3.connection.request.timeout"), + Map.entry("gs.connection.timeout", "s3.connection.timeout"), + Map.entry("gs.use_path_style", "use_path_style"), + Map.entry("gs.force_parsing_by_standard_uri", "force_parsing_by_standard_uri")); + + @Override + public Map normalizeProperties(Map properties, Map context) { + GcsAuthResolver.resolve(properties); + Map normalized = new HashMap<>(properties); + String selected = context.get("provider"); + if (selected == null || selected.isBlank()) { + if (!GcsAuthResolver.guessIsGcs(context)) { + return normalized; + } + selected = "GCP"; + normalized.put("provider", selected); + } + if ("GCP".equalsIgnoreCase(selected)) { + RESOURCE_ALIASES.forEach((alias, key) -> { + String value = normalized.remove(alias); + if (value != null && !value.isBlank()) { + normalized.put(key, value); + } + }); + } + return normalized; + } + + @Override + public Optional resolveAuthentication(Map properties) { + return GcsAuthResolver.resolve(properties).map(auth -> { + Map credential = new HashMap<>(); + Map updates = new HashMap<>(); + auth.getNativeCredential().ifPresent(nativeCredential -> { + credential.put("credential_provider_type", nativeCredential.getCredentialProviderType().name()); + if (!nativeCredential.getImpersonationServiceAccount().isEmpty()) { + credential.put("impersonation_service_account", nativeCredential.getImpersonationServiceAccount()); + } + if (properties.containsKey(GcpCredential.CREDENTIAL_PROVIDER_TYPE)) { + updates.put("credential_provider_type", nativeCredential.getCredentialProviderType().name()); + } + if (properties.containsKey(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)) { + updates.put("impersonation_service_account", nativeCredential.getImpersonationServiceAccount()); + } + }); + return new ObjectStorageAuthentication("GCP", auth.isAnonymous(), credential, updates); + }); + } + + @Override + public Map credentialToProperties(Map credential) { + Map properties = new HashMap<>(); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.get("credential_provider_type")); + String principal = credential.get("impersonation_service_account"); + if (principal != null && !principal.isEmpty()) { + properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, principal); + } + return properties; + } + + @Override + public Set modifiableCredentialPropertyKeys() { + return Set.of(GcpCredential.CREDENTIAL_PROVIDER_TYPE, GcpCredential.IMPERSONATION_SERVICE_ACCOUNT); + } + + @Override + public Set clearablePropertyKeys() { + return Set.of(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT); + } + @Override public boolean supports(Map properties) { String providerValue = properties.get(S3CompatSignals.PROVIDER_KEY); diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java index a79261c0d9b79f..bd93d4d0d19990 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcpAuthTest.java @@ -17,8 +17,8 @@ package org.apache.doris.filesystem.gcs; -import org.apache.doris.filesystem.auth.GcpCredential; -import org.apache.doris.filesystem.auth.GcpCredentialProviderType; +import org.apache.doris.filesystem.gcs.auth.GcpCredential; +import org.apache.doris.filesystem.gcs.auth.GcpCredentialProviderType; import com.google.auth.oauth2.AccessToken; import com.google.auth.oauth2.ComputeEngineCredentials; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java index cf1891e809e3d7..022618003d4f92 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java @@ -18,6 +18,7 @@ package org.apache.doris.filesystem.gcs; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.s3.S3FileSystem; import org.apache.doris.filesystem.s3.S3ObjStorage; @@ -212,4 +213,70 @@ void supportsGuess_respectsExplicitS3AndRecognizesRegionalEndpoints() { props.remove("provider"); } } + + @Test + void normalizePropertiesUsesEffectiveProviderForPartialAlter() { + Map stored = Map.of("provider", "GCP", "s3.endpoint", "old.endpoint"); + Map patch = Map.of("gs.endpoint", "storage.googleapis.com", + "gs.connection.maximum", "25", "gs.impersonation_service_account", ""); + Map normalized = provider.normalizeProperties(patch, stored); + Assertions.assertEquals(Map.of("s3.endpoint", "storage.googleapis.com", "s3.connection.maximum", "25", + "gs.impersonation_service_account", ""), normalized); + Assertions.assertEquals("old.endpoint", stored.get("s3.endpoint")); + Assertions.assertTrue(patch.containsKey("gs.endpoint")); + Assertions.assertTrue(provider.clearablePropertyKeys().contains("gs.impersonation_service_account")); + Assertions.assertTrue(provider.modifiableCredentialPropertyKeys().contains("gs.credential_provider_type")); + Assertions.assertFalse(provider.modifiableCredentialPropertyKeys().contains("s3.endpoint")); + } + + @Test + void normalizePropertiesPreservesSelectionAndValidatesRawAliases() { + Map inferred = Map.of("gs.endpoint", "storage.googleapis.com"); + Assertions.assertEquals(Map.of("provider", "GCP", "s3.endpoint", "storage.googleapis.com"), + provider.normalizeProperties(inferred, inferred)); + Map s3 = Map.of("provider", "S3", "s3.endpoint", "storage.googleapis.com"); + Assertions.assertEquals(s3, provider.normalizeProperties(s3, s3)); + Assertions.assertTrue(provider.resolveAuthentication(s3).isEmpty()); + Map conflict = Map.of("gs.credential_provider_type", "DEFAULT", + "gs.access_key", "ak", "gs.secret_key", "sk"); + Assertions.assertThrows(IllegalArgumentException.class, + () -> provider.normalizeProperties(conflict, conflict)); + } + + @Test + void nativeAuthenticationSeparatesCreateDefaultsFromExplicitAlterFields() { + ObjectStorageAuthentication defaults = provider.resolveAuthentication(Map.of("provider", "GCP")).get(); + Assertions.assertEquals("GCP", defaults.getProvider()); + Assertions.assertTrue(defaults.isNative()); + Assertions.assertFalse(defaults.isAnonymous()); + Assertions.assertEquals(Map.of("credential_provider_type", "DEFAULT"), defaults.getCredential()); + Assertions.assertTrue(defaults.getCredentialUpdates().isEmpty()); + + ObjectStorageAuthentication clear = provider.resolveAuthentication( + Map.of("gs.impersonation_service_account", "")).get(); + Assertions.assertEquals(defaults.getCredential(), clear.getCredential()); + Assertions.assertEquals(Map.of("impersonation_service_account", ""), clear.getCredentialUpdates()); + Assertions.assertFalse(clear.getCredentialUpdates().containsKey("credential_provider_type")); + + Map properties = Map.of("gs.credential_provider_type", "COMPUTE_ENGINE", + "gs.impersonation_service_account", "reader@project.iam.gserviceaccount.com"); + ObjectStorageAuthentication explicit = provider.resolveAuthentication(properties).get(); + Assertions.assertEquals(explicit.getCredential(), explicit.getCredentialUpdates()); + Assertions.assertEquals(properties, provider.credentialToProperties(explicit.getCredential())); + } + + @Test + void hmacAndAnonymousDoNotPublishNativeWireCredentials() { + ObjectStorageAuthentication hmac = provider.resolveAuthentication(Map.of( + "provider", "GCP", "gs.access_key", "ak", "gs.secret_key", "sk")).get(); + Assertions.assertFalse(hmac.isNative()); + Assertions.assertFalse(hmac.isAnonymous()); + Assertions.assertTrue(hmac.getCredentialUpdates().isEmpty()); + ObjectStorageAuthentication anonymous = provider.resolveAuthentication(Map.of( + "provider", "GCP", "gs.credential_provider_type", "ANONYMOUS")).get(); + Assertions.assertTrue(anonymous.isAnonymous()); + Assertions.assertFalse(anonymous.isNative()); + Assertions.assertTrue(anonymous.getCredentialUpdates().isEmpty()); + } + } diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml b/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml index ae5ef4de32e4db..78202d93f10138 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml @@ -37,6 +37,11 @@ under the License. Not a plugin itself: no provider, no META-INF/services. + + org.apache.doris + fe-filesystem-gcs-common + ${revision} + org.apache.doris fe-filesystem-spi diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java index b158f88a123462..1f42ad43528b74 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java @@ -180,7 +180,7 @@ public static boolean guessAllowed(Map properties) { * key comparison is genuinely case-insensitive, which is what legacy clearly intended. */ public static boolean guessIsGcs(Map properties) { - return org.apache.doris.filesystem.auth.GcsAuthResolver.guessIsGcs(properties); + return org.apache.doris.filesystem.gcs.auth.GcsAuthResolver.guessIsGcs(properties); } /** diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java index ecb5731d2718ea..e565954620ed1a 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java @@ -208,8 +208,8 @@ public static S3FileSystemProperties of(Map properties) { @Override public void validate() { - if (org.apache.doris.filesystem.auth.GcsAuthResolver.hasNativeCredentialProperties(rawProperties)) { - org.apache.doris.filesystem.auth.GcsAuthResolver.resolve(rawProperties); + if (org.apache.doris.filesystem.gcs.auth.GcsAuthResolver.hasNativeCredentialProperties(rawProperties)) { + org.apache.doris.filesystem.gcs.auth.GcsAuthResolver.resolve(rawProperties); throw new IllegalArgumentException("Native GCP authentication cannot be used with S3 properties."); } new ParamRules() diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java index 7e4859dbb147a5..d4bf8e374fa7e0 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java @@ -20,13 +20,16 @@ import org.apache.doris.extension.spi.Plugin; import org.apache.doris.extension.spi.PluginFactory; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.FileSystemCapability; import org.apache.doris.filesystem.properties.FileSystemProperties; import java.io.IOException; import java.util.Collections; import java.util.EnumSet; +import java.util.HashMap; import java.util.Map; +import java.util.Optional; import java.util.Set; /** @@ -44,6 +47,35 @@ */ public interface FileSystemProvider

extends PluginFactory { + /** + * Normalize provider-owned aliases without binding clients or requiring a complete property map. + * Context is the effective CREATE/ALTER selection; properties may be a partial ALTER patch. + * Implementations return a new map and preserve explicit credential-clearing values. + */ + default Map normalizeProperties(Map properties, Map context) { + return new HashMap<>(properties); + } + + /** Resolve authentication without I/O, including validation of conflicting raw aliases. */ + default Optional resolveAuthentication(Map properties) { + return Optional.empty(); + } + + /** Translate protocol-native fields back into provider-owned user properties. */ + default Map credentialToProperties(Map credential) { + throw new UnsupportedOperationException(name() + " has no native credential property codec"); + } + + /** Provider-owned credential properties accepted by a storage vault ALTER. */ + default Set modifiableCredentialPropertyKeys() { + return Collections.emptySet(); + } + + /** Properties whose explicit empty ALTER value clears the persisted setting. */ + default Set clearablePropertyKeys() { + return Collections.emptySet(); + } + /** * Returns true if this provider can handle the given properties. * Must be cheap (no network calls) and deterministic. diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java index 16c91831c14485..5e44a0866db6f2 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java @@ -68,6 +68,7 @@ public class FileSystemPluginSurfaceTest { /** The types a filesystem plugin implements or calls. Everything reachable on them is the contract. */ private static final List> FROZEN_TYPES = Arrays.asList( FileSystemProvider.class, + org.apache.doris.filesystem.auth.ObjectStorageAuthentication.class, ObjFileSystem.class, ObjStorage.class, org.apache.doris.extension.spi.Plugin.class, diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt index 819b07ca74430e..f88e7f19ea852a 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt @@ -5,15 +5,25 @@ org.apache.doris.extension.spi.PluginFactory#create():org.apache.doris.extension org.apache.doris.extension.spi.PluginFactory#create(org.apache.doris.extension.spi.PluginContext):org.apache.doris.extension.spi.Plugin org.apache.doris.extension.spi.PluginFactory#description():java.lang.String org.apache.doris.extension.spi.PluginFactory#name():java.lang.String +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getCredential():java.util.Map +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getCredentialUpdates():java.util.Map +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getProvider():java.lang.String +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#isAnonymous():boolean +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#isNative():boolean org.apache.doris.filesystem.spi.FileSystemProvider#bind(java.util.Map):org.apache.doris.filesystem.properties.FileSystemProperties org.apache.doris.filesystem.spi.FileSystemProvider#capabilities(org.apache.doris.filesystem.properties.FileSystemProperties):java.util.Set +org.apache.doris.filesystem.spi.FileSystemProvider#clearablePropertyKeys():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#create():org.apache.doris.extension.spi.Plugin org.apache.doris.filesystem.spi.FileSystemProvider#create(java.util.Map):org.apache.doris.filesystem.FileSystem org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.extension.spi.PluginContext):org.apache.doris.extension.spi.Plugin org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem org.apache.doris.filesystem.spi.FileSystemProvider#createUntyped(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem +org.apache.doris.filesystem.spi.FileSystemProvider#credentialToProperties(java.util.Map):java.util.Map org.apache.doris.filesystem.spi.FileSystemProvider#description():java.lang.String +org.apache.doris.filesystem.spi.FileSystemProvider#modifiableCredentialPropertyKeys():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#name():java.lang.String +org.apache.doris.filesystem.spi.FileSystemProvider#normalizeProperties(java.util.Map,java.util.Map):java.util.Map +org.apache.doris.filesystem.spi.FileSystemProvider#resolveAuthentication(java.util.Map):java.util.Optional org.apache.doris.filesystem.spi.FileSystemProvider#sensitivePropertyKeys():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#supports(java.util.Map):boolean org.apache.doris.filesystem.spi.FileSystemProvider#supportsExplicit(java.util.Map):boolean diff --git a/fe/fe-filesystem/pom.xml b/fe/fe-filesystem/pom.xml index a8368afa785571..73a8e6bf4a96f5 100644 --- a/fe/fe-filesystem/pom.xml +++ b/fe/fe-filesystem/pom.xml @@ -53,12 +53,13 @@ under the License. bumping all five families. See plan-doc/designs/2026-07-29-plugin-api-version-check-design.md. --> - 1.0 + 2.0 fe-filesystem-api fe-filesystem-spi + fe-filesystem-gcs-common fe-filesystem-s3-base fe-filesystem-s3 fe-filesystem-gcs From 131e521558fdedbe9f47f48693120ea0b2893629 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Sat, 10 Oct 2026 14:35:18 +0800 Subject: [PATCH 26/47] [fix](iceberg) Pass storage Hadoop configuration to Glue FileIO ### What problem does this PR solve? Related PR: #68540 Problem Summary: Glue catalog creation discarded the bound storage Hadoop configuration. With native GCP authentication, ResolvingFileIO falls back to HadoopFileIO for gs:// locations and therefore lost the selected credential source and impersonation account. Use the existing configuration builder for Glue, preserving plugin classloader pinning and storage property overrides. ### Release note Iceberg Glue catalogs now pass native GCS authentication settings to HadoopFileIO. ### Check List (For Author) - Test: Added connector-to-GlueCatalog-to-FileIO coverage for DEFAULT and COMPUTE_ENGINE credentials, with and without impersonation. The test reads a local fixture through gs:// Hadoop fallback without cloud API calls. - Test: Iceberg Checkstyle, connector import gate and git diff --check passed. Compilation and unit tests were not run: thirdparty/installed/bin/protoc is absent from this checkout. - Behavior changed: Glue FileIO receives bound Hadoop configuration instead of null; existing S3FileIO continues to use its catalog options. - Does this need documentation: No new properties or API/protocol changes. - Self-review: The change stays within the Iceberg plugin and uses the existing configuration and classloader path. No new threads, locks, resource ownership, persistence changes or network operations are introduced. Regression coverage exercises the actual catalog branch missed by the previous factory-only tests. --- .../connector/iceberg/IcebergConnector.java | 7 +- .../IcebergConnectorGlueFileIOTest.java | 110 ++++++++++++++++++ 2 files changed, 112 insertions(+), 5 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorGlueFileIOTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index da37ab2063987d..274c92f8fc8a73 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -1040,16 +1040,13 @@ private Catalog createCatalog() { catalogOptions.get("client-pool-cache-keys"))); break; } - case IcebergCatalogProperties.TYPE_GLUE: - // Legacy IcebergGlueMetaStoreProperties builds the catalog with conf=null. - conf = null; - break; case IcebergCatalogProperties.TYPE_JDBC: maybeRegisterJdbcDriver(); conf = IcebergCatalogFactory.buildHadoopConfiguration(properties, storageHadoopConfig); break; default: - // rest / hadoop: a storage Configuration from the fe-filesystem-bound storage + raw + // rest / hadoop / glue: FileIO needs the bound storage configuration, including + // native GCS credentials when ResolvingFileIO falls back to HadoopFileIO, plus raw // fs./dfs./hadoop. passthrough. conf = IcebergCatalogFactory.buildHadoopConfiguration(properties, storageHadoopConfig); break; diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorGlueFileIOTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorGlueFileIOTest.java new file mode 100644 index 00000000000000..2fb33077bf0a50 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorGlueFileIOTest.java @@ -0,0 +1,110 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.filesystem.gcs.GcsFileSystemProperties; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.RawLocalFileSystem; +import org.apache.iceberg.TableOperations; +import org.apache.iceberg.aws.glue.GlueCatalog; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.hadoop.HadoopInputFile; +import org.apache.iceberg.io.FileIO; +import org.apache.iceberg.io.ResolvingFileIO; +import org.apache.iceberg.io.SeekableInputStream; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +class IcebergConnectorGlueFileIOTest { + + @Test + void gluePassesNativeGcsConfigurationToHadoopFileIO(@TempDir java.nio.file.Path temp) throws Exception { + java.nio.file.Path metadata = temp.resolve("metadata.json"); + Files.writeString(metadata, "metadata", StandardCharsets.UTF_8); + for (String source : List.of("DEFAULT", "COMPUTE_ENGINE")) { + for (String account : List.of("", "reader@test.iam.gserviceaccount.com")) { + Map properties = new HashMap<>(Map.of( + "iceberg.catalog.type", "glue", + "provider", "GCP", + "warehouse", "gs://test-bucket/warehouse", + "gs.credential_provider_type", source, + "gs.impersonation_service_account", account, + "glue.access_key", "test-ak", + "glue.secret_key", "test-sk", + "glue.region", "us-east-1", + "glue.endpoint", "http://127.0.0.1:1")); + // Replace only the Hadoop transport. Keep the real connector, GlueCatalog, + // table operations and ResolvingFileIO -> HadoopFileIO dispatch. + properties.put("fs.gs.impl", LocalGcsFileSystem.class.getName()); + properties.put("fs.gs.impl.disable.cache", "true"); + RecordingConnectorContext context = new RecordingConnectorContext(); + context.storageProperties = List.of(GcsFileSystemProperties.of(properties)); + try (IcebergConnector connector = new IcebergConnector(properties, context)) { + connector.getMetadata(null); + Field catalogField = IcebergConnector.class.getDeclaredField("icebergCatalog"); + catalogField.setAccessible(true); + GlueCatalog catalog = (GlueCatalog) catalogField.get(connector); + // Construct table IO without contacting Glue to fetch table metadata. + Method newTableOps = GlueCatalog.class.getDeclaredMethod("newTableOps", TableIdentifier.class); + newTableOps.setAccessible(true); + TableOperations operations = (TableOperations) newTableOps.invoke( + catalog, TableIdentifier.of("db", "table")); + FileIO io = operations.io(); + Assertions.assertInstanceOf(ResolvingFileIO.class, io); + Configuration conf = ((ResolvingFileIO) io).getConf(); + Assertions.assertNotNull(conf, "Glue must receive the bound storage Hadoop configuration"); + String authType = "DEFAULT".equals(source) ? "APPLICATION_DEFAULT" : "COMPUTE_ENGINE"; + Assertions.assertEquals(authType, conf.get("fs.gs.auth.type")); + Assertions.assertEquals(account, conf.get("fs.gs.auth.impersonation.service.account")); + Assertions.assertEquals("https://storage.googleapis.com/", conf.get("fs.gs.storage.root.url")); + + HadoopInputFile input = Assertions.assertInstanceOf(HadoopInputFile.class, + io.newInputFile(new Path("gs", "test-bucket", metadata.toString()).toString())); + try (FileSystem fs = input.getFileSystem(); SeekableInputStream stream = input.newStream()) { + Assertions.assertInstanceOf(LocalGcsFileSystem.class, fs); + Assertions.assertEquals(authType, fs.getConf().get("fs.gs.auth.type")); + Assertions.assertEquals(account, + fs.getConf().get("fs.gs.auth.impersonation.service.account")); + Assertions.assertEquals("metadata", new String(stream.readAllBytes(), StandardCharsets.UTF_8)); + } + } + } + } + } + + /** Serves gs://test-bucket paths from the local test directory without cloud credentials or I/O. */ + public static class LocalGcsFileSystem extends RawLocalFileSystem { + @Override + public URI getUri() { + return URI.create("gs://test-bucket/"); + } + } +} From d8de5797b124d385e89569f56ed4713fae2245cf Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Sat, 10 Oct 2026 16:34:23 +0800 Subject: [PATCH 27/47] [fix](storage) Preserve legacy GCP credentials and discard inactive aliases ### What problem does this PR solve? Related PR: #68540 Problem Summary: GCP resources explicitly selecting the legacy AWS DEFAULT credential chain without static keys were misclassified as native ADC, breaking CREATE, ALTER and policy publication after replay. Preserve that HMAC/default-chain selection and the plugin's backend credential mode, and always release resource read locks if policy serialization fails. Inactive gs.* aliases could also survive an S3 ALTER and unexpectedly overwrite the endpoint or credentials on a subsequent provider-only ALTER. Remove inactive aliases and normalize persisted properties using their original provider before applying updates. Finally, initialize ca_cert_file_path in the signed URL test: BE UT build 1065840 stopped at compilation with -Wmissing-designated-field-initializers. ### Release note Preserve legacy GCP default-chain authentication and prevent provider changes from activating stale resource aliases. ### Check List (For Author) - Test: Added resolver conflict/default-chain coverage and resource tests for CREATE/ALTER, serialization replay, policy push, exception lock release and two-step ALTER with legacy dormant aliases. - Test: Offline Checkstyle passed for fe-filesystem-gcs-common, fe-filesystem-gcs and fe-core; C++ check-format, BE build hygiene, FE connector-import and metadata-funnel gates, and git diff --check passed. - Test: FE and BE unit tests were not executed locally because thirdparty/installed (including protoc) and BE test build artifacts are unavailable. The CI compile fix matches the downloaded build 1065840 diagnostic; CI rerun remains necessary. - Behavior changed: Yes; preserve explicit legacy DEFAULT authentication, discard inactive resource aliases and release locks on failed policy publication. - Does this need documentation: No; compatibility and build fixes introduce no new configuration. - Self-review (goal/scope): Covers review findings 1 and 2 and the reported BE compile failure. GCP interpretation remains in the filesystem plugin; core only preserves plugin output and generic resource state. - Self-review (concurrency/lifecycle): Existing synchronized ALTER and write-lock publication are retained. Resource serialization now pairs readLock with finally/readUnlock, matching policy serialization. No new lifecycle or static initialization dependencies. - Self-review (compatibility/persistence/protocol): Resource serialization and wire schemas are unchanged. Replayed aliases use their original provider; native selectors still reject AWS credential selectors. Both policy Thrift and backend property paths preserve legacy default-chain behavior. - Self-review (coverage/observability/performance): Tests cover successful replay and invalid authentication, plus new and already-persisted dormant aliases. No new network calls, logging, metrics, memory ownership, transaction semantics or asymptotic cost are introduced. --- .../io/fs/gcs_signed_url_provider_test.cpp | 3 +- .../org/apache/doris/catalog/S3Resource.java | 3 +- .../datasource/storage/StorageAdapter.java | 6 +- .../doris/task/PushStoragePolicyTask.java | 19 ++-- .../apache/doris/catalog/S3ResourceTest.java | 90 +++++++++++++++++++ .../filesystem/gcs/auth/GcsAuthResolver.java | 9 +- .../gcs/auth/GcsAuthResolverTest.java | 19 ++++ .../filesystem/gcs/GcsFileSystemProvider.java | 4 + 8 files changed, 136 insertions(+), 17 deletions(-) diff --git a/be/test/io/fs/gcs_signed_url_provider_test.cpp b/be/test/io/fs/gcs_signed_url_provider_test.cpp index 6d4e17a72c991c..503cefea9fff89 100644 --- a/be/test/io/fs/gcs_signed_url_provider_test.cpp +++ b/be/test/io/fs/gcs_signed_url_provider_test.cpp @@ -73,7 +73,8 @@ TEST_F(GcsSignedUrlProviderTest, DefaultSignsErrorLogWithResolvedIdentity) { GcsV4SignedUrlProviderOptions options {.endpoint = "storage.googleapis.com", .bucket = "bucket", .key = "load-errors/error.log", - .expiration_secs = 300}; + .expiration_secs = 300, + .ca_cert_file_path = ""}; std::string url; auto status = generate_gcs_v4_signed_url(options, credential, tokens, &url); ASSERT_TRUE(status.ok()) << status; diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index dca93dae845dd5..e21faeef95fdb9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -234,8 +234,9 @@ public synchronized void modifyProperties(Map newProperties) thr // Normalize the patch independently so its aliases win over persisted canonical values. S3ResourceCompat.convertToStdProperties(properties); Map normalizedUpdates = StorageAdapter.normalizeProperties(properties, selectionProperties); + // Interpret persisted aliases using their original provider before applying a provider change. Map effectiveProperties = - StorageAdapter.normalizeProperties(this.properties, selectionProperties); + StorageAdapter.normalizeProperties(this.properties, this.properties); S3ResourceCompat.convertToStdProperties(effectiveProperties); for (Map.Entry update : normalizedUpdates.entrySet()) { // Empty updates are ignored, except when clearing a session token or impersonation account. diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index 4e726f2780f328..35b547960b8a7a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -544,15 +544,15 @@ private Map alignS3FamilyBackendMap(S3CompatibleFileSystemProper // AWS_EXTERNAL_ID pass through unconditionally when set (the SPI already emits them). aligned.put("AWS_CREDENTIALS_PROVIDER_TYPE", s3CredentialsMode.getMode()); } else { - // Align fe-core, ledger 2.4-5: non-S3 dialects never emit role keys, and emit - // AWS_CREDENTIALS_PROVIDER_TYPE=ANONYMOUS exactly when both AK and SK are blank. + // Non-S3 dialects never emit role keys. Preserve an explicit credential mode + // supplied by the filesystem plugin when no static or native credentials are present. aligned.remove("AWS_ROLE_ARN"); aligned.remove("AWS_EXTERNAL_ID"); if (s3.hasStaticCredentials() || resolveAuthentication(aligned).map(ObjectStorageAuthentication::isNative).orElse(false)) { aligned.remove("AWS_CREDENTIALS_PROVIDER_TYPE"); } else { - aligned.put("AWS_CREDENTIALS_PROVIDER_TYPE", AwsCredentialsProviderMode.ANONYMOUS.name()); + aligned.putIfAbsent("AWS_CREDENTIALS_PROVIDER_TYPE", AwsCredentialsProviderMode.ANONYMOUS.name()); } } return aligned; diff --git a/fe/fe-core/src/main/java/org/apache/doris/task/PushStoragePolicyTask.java b/fe/fe-core/src/main/java/org/apache/doris/task/PushStoragePolicyTask.java index 0b9a51b39948ef..63cee8f9a9e67f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/task/PushStoragePolicyTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/task/PushStoragePolicyTask.java @@ -84,15 +84,18 @@ public TPushStoragePolicyReq toThrift() { resource.forEach(r -> { TStorageResource item = new TStorageResource(); r.readLock(); - item.setId(r.getId()); - item.setName(r.getName()); - item.setVersion(r.getVersion()); - if (r.getType() == ResourceType.S3) { - item.setS3StorageParam(S3ThriftAdapter.getS3TStorageParam(r.getCopiedProperties())); - } else if (r.getType() == ResourceType.HDFS) { - item.setHdfsStorageParam(HdfsResource.generateHdfsParam(r.getCopiedProperties())); + try { + item.setId(r.getId()); + item.setName(r.getName()); + item.setVersion(r.getVersion()); + if (r.getType() == ResourceType.S3) { + item.setS3StorageParam(S3ThriftAdapter.getS3TStorageParam(r.getCopiedProperties())); + } else if (r.getType() == ResourceType.HDFS) { + item.setHdfsStorageParam(HdfsResource.generateHdfsParam(r.getCopiedProperties())); + } + } finally { + r.readUnlock(); } - r.readUnlock(); tStorageResources.add(item); }); ret.setResource(tStorageResources); diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 0a4174326ee985..1456b263f115c6 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -32,6 +32,7 @@ import org.apache.doris.nereids.trees.plans.commands.CreateResourceCommand; import org.apache.doris.nereids.trees.plans.commands.info.CreateResourceInfo; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.task.PushStoragePolicyTask; import org.apache.doris.thrift.TGcpCredential; import org.apache.doris.thrift.TS3StorageParam; @@ -55,6 +56,7 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; @@ -635,6 +637,92 @@ public void testGsEndpointUsesEffectiveProviderOnAlter() throws Exception { Assertions.assertEquals("https://custom.example.com", resource.getProperty(S3ResourceCompat.ENDPOINT)); } + @Test + public void testInactiveGsAliasesCannotReappearAfterProviderAlter() throws Exception { + for (boolean replayLegacyAliases : new boolean[] {false, true}) { + S3Resource resource = new S3Resource("s3_resource"); + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "S3"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Map aliases = ImmutableMap.of("gs.endpoint", "https://ignored.example.com", + "gs.access_key", "ignored-access", "gs.secret_key", "ignored-secret"); + if (replayLegacyAliases) { + // Simulate the dormant aliases persisted by an older FE. + Map stored = resource.getCopiedProperties(); + stored.putAll(aliases); + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + field.set(resource, stored); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (DataOutputStream output = new DataOutputStream(bytes)) { + resource.write(output); + } + try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + resource = (S3Resource) Resource.read(input); + } + } else { + resource.modifyProperties(aliases); + Assertions.assertFalse(resource.getCopiedProperties().keySet().stream() + .anyMatch(key -> key.startsWith("gs."))); + } + resource.modifyProperties(ImmutableMap.of("provider", "GCP")); + TS3StorageParam thrift = S3ThriftAdapter.getS3TStorageParam(resource.getCopiedProperties()); + Assertions.assertEquals(s3Endpoint, thrift.getEndpoint()); + Assertions.assertEquals(s3AccessKey, thrift.getAk()); + Assertions.assertEquals(s3SecretKey, thrift.getSk()); + Assertions.assertFalse(resource.getCopiedProperties().keySet().stream() + .anyMatch(key -> key.startsWith("gs."))); + } + } + + @Test + public void testLegacyGcpDefaultChainSurvivesReplayAndPolicyPush() throws Exception { + for (String key : new String[] {"s3.credentials_provider_type", "AWS_CREDENTIALS_PROVIDER_TYPE"}) { + Map properties = new HashMap<>(s3Properties); + properties.remove("AWS_ACCESS_KEY"); + properties.remove("AWS_SECRET_KEY"); + properties.put("provider", "GCP"); + properties.put(key, "DEFAULT"); + S3Resource resource = new S3Resource("gcp_default_chain"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.CONNECTION_TIMEOUT_MS, "2000")); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (DataOutputStream output = new DataOutputStream(bytes)) { + resource.write(output); + } + S3Resource restored; + try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + restored = (S3Resource) Resource.read(input); + } + Assertions.assertEquals(resource.getCopiedProperties(), restored.getCopiedProperties()); + Assertions.assertEquals("DEFAULT", StorageAdapter.of(restored.getCopiedProperties()) + .getBackendConfigProperties().get("AWS_CREDENTIALS_PROVIDER_TYPE")); + PushStoragePolicyTask task = new PushStoragePolicyTask(1L, List.of(), List.of(restored), List.of()); + TS3StorageParam thrift = task.toThrift().getResource().get(0).getS3StorageParam(); + Assertions.assertFalse(thrift.isSetCredential()); + Assertions.assertFalse(thrift.isSetCredProviderType()); + Assertions.assertEquals(2000, thrift.getConnTimeoutMs()); + } + } + + @Test + public void testPolicyPushReleasesResourceLockOnInvalidAuthentication() throws Exception { + S3Resource resource = new S3Resource("invalid_gcp_resource"); + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + resource.setProperties(ImmutableMap.copyOf(properties)); + Map invalid = resource.getCopiedProperties(); + invalid.put("gs.credential_provider_type", "DEFAULT"); + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + field.set(resource, invalid); + S3Resource tracked = Mockito.spy(resource); + PushStoragePolicyTask task = new PushStoragePolicyTask(1L, List.of(), List.of(tracked), List.of()); + Assertions.assertThrows(IllegalArgumentException.class, task::toThrift); + Mockito.verify(tracked).readLock(); + Mockito.verify(tracked).readUnlock(); + } + @Test public void testGsAliasesMatchConnectorAndResourceProtocols() throws Exception { for (String provider : new String[] {"GCP", "gcp"}) { @@ -731,6 +819,8 @@ public void testGsAliasesDoNotOverrideOtherProviders() throws Exception { Assertions.assertEquals(s3SecretKey, thrift.getSk()); Assertions.assertEquals(Integer.parseInt(s3ConnTimeoutMs), thrift.getConnTimeoutMs()); Assertions.assertFalse(thrift.isUsePathStyle()); + Assertions.assertFalse(resource.getCopiedProperties().keySet().stream() + .anyMatch(key -> key.startsWith("gs."))); } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java index d873f786717fac..e0b7b34aece015 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java @@ -44,7 +44,7 @@ public static Optional resolve(Map properties) { || "true".equalsIgnoreCase(getPropertyIgnoreCase(properties, "fs.gcs.support")); boolean hasAccessKey = hasNonBlankProperty(properties, ACCESS_KEYS); boolean hasSecretKey = hasNonBlankProperty(properties, SECRET_KEYS); - boolean hasLegacyAnonymous = hasLegacyAnonymousProvider(properties); + boolean hasLegacyAnonymous = hasLegacyProvider(properties, "ANONYMOUS"); String provider = getPropertyIgnoreCase(properties, "provider"); if (isNotBlank(provider) && !"GCP".equalsIgnoreCase(provider) && !"GCS".equalsIgnoreCase(provider)) { if (hasNativeProperties) { @@ -73,7 +73,8 @@ public static Optional resolve(Map properties) { : type == GcpCredentialProviderType.COMPUTE_ENGINE ? GcsAuth.Mode.COMPUTE_ENGINE : GcsAuth.Mode.ADC; } else if (hasLegacyAnonymous) { mode = GcsAuth.Mode.ANONYMOUS; - } else if (hasAccessKey || hasSecretKey) { + } else if (hasAccessKey || hasSecretKey || hasLegacyProvider(properties, "DEFAULT")) { + // Explicit legacy DEFAULT uses ambient HMAC credentials through the AWS default chain. mode = GcsAuth.Mode.HMAC; } else { mode = GcsAuth.Mode.ADC; @@ -138,9 +139,9 @@ private static String trim(String value) { return value == null ? null : value.trim(); } - private static boolean hasLegacyAnonymousProvider(Map properties) { + private static boolean hasLegacyProvider(Map properties, String providerType) { for (String name : AWS_PROVIDERS) { - if ("ANONYMOUS".equalsIgnoreCase(trim(getPropertyIgnoreCase(properties, name)))) { + if (providerType.equalsIgnoreCase(trim(getPropertyIgnoreCase(properties, name)))) { return true; } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java index c07b730b00c576..f056848fee097c 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java @@ -105,6 +105,25 @@ public void testExplicitS3OmittedAuthenticationPreservesDefaultChain() { Assertions.assertEquals(GcsAuth.Mode.ADC, GcsAuthResolver.resolve(props).get().getMode()); } + @Test + public void testLegacyGcpDefaultChainDoesNotAcquireAdc() { + for (String key : new String[] {"s3.credentials_provider_type", "AWS_CREDENTIALS_PROVIDER_TYPE"}) { + Map props = new HashMap<>(); + props.put("provider", "GCP"); + props.put(key, " default "); + GcsAuth auth = GcsAuthResolver.resolve(props).orElseThrow(); + Assertions.assertEquals(GcsAuth.Mode.HMAC, auth.getMode()); + Assertions.assertFalse(auth.getNativeCredential().isPresent()); + Assertions.assertFalse(auth.isAnonymous()); + // Native selectors still conflict with an explicitly selected AWS credential chain. + props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, "DEFAULT"); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + props.remove(GcpCredential.CREDENTIAL_PROVIDER_TYPE); + props.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "target@test.iam.gserviceaccount.com"); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + } + } + @Test public void testOtherProvidersAndConflictingNativeProperties() { for (String provider : new String[] {"S3", "AZURE", "OSS"}) { diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index 13aeca870a5560..00fe30bb105a5f 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -66,6 +66,7 @@ public Map normalizeProperties(Map properties, M String selected = context.get("provider"); if (selected == null || selected.isBlank()) { if (!GcsAuthResolver.guessIsGcs(context)) { + normalized.keySet().removeAll(RESOURCE_ALIASES.keySet()); return normalized; } selected = "GCP"; @@ -78,6 +79,9 @@ public Map normalizeProperties(Map properties, M normalized.put(key, value); } }); + } else { + // Inactive aliases must not become effective after a later provider-only ALTER. + normalized.keySet().removeAll(RESOURCE_ALIASES.keySet()); } return normalized; } From 50e84be66db3bd41c6c912e118d8ccdf7107af5b Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 12 Oct 2026 11:12:55 +0800 Subject: [PATCH 28/47] [fix](test) Avoid closing an uninitialized GCS filesystem ### What problem does this PR solve? Related PR: #68540 Problem Summary: IcebergPluginTest only constructs GoogleHadoopFileSystem by reflection to verify that the native GCS runtime is available. The try-with-resources block closes it without initialize(), so Hadoop's FileSystem.close() calls getUri() while fsRoot is still null. Keep the class resolution and construction assertions without closing this uninitialized instance. Construction starts no background tasks or GCS clients, so the test does not need initialized filesystem cleanup. ### Release note None ### Check List (For Author) - Test: Reproduced the original fsRoot NPE with the extracted test method and cached Hadoop 3.4.2 / GCS connector 3.1.18 dependencies on JDK 20; the fixed method passes. Module Checkstyle and git diff --check passed. run-fe-ut.sh could not start because the configured JDK 17 cannot initialize its VM; the complete FE suite was not run. - Behavior changed: No (test lifecycle only) - Does this need documentation: No --- .../java/org/apache/doris/iceberg/IcebergPluginTest.java | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java b/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java index cf7ce236cc753b..7965c280b42f21 100644 --- a/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java +++ b/fe/be-java-extensions/iceberg-metadata-scanner/src/test/java/org/apache/doris/iceberg/IcebergPluginTest.java @@ -117,10 +117,11 @@ public void reconstructsNativeGcpFileIoAfterSerialization() throws Exception { public void instantiatesNativeGcsFilesystem() throws Exception { Configuration conf = new Configuration(); conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); - try (FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance()) { - Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", - fs.getClass().getName()); - } + // This only checks construction. No resources are opened until initialize(), and close() + // requires the filesystem root that initialize() sets up. + FileSystem fs = FileSystem.getFileSystemClass("gs", conf).getDeclaredConstructor().newInstance(); + Assertions.assertEquals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem", + fs.getClass().getName()); } @Test From 56740709057bea83bba1a4df07fe6115c279e0a5 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 12 Oct 2026 15:03:13 +0800 Subject: [PATCH 29/47] [refactor](fe) Move native credential wire conversion into filesystem plugins ### What problem does this PR solve? Related PR: #68540 Problem Summary: Native GCP authentication still required a GCP-specific adapter and a hard-coded factory branch in fe-core. Introduce an optional ObjectStorageCredentialCodec on filesystem providers and implement the GCP wire conversion in the GCS plugin. The core discovers codecs from the existing provider registry and transfers complete credential envelopes without interpreting provider-specific fields. Cloud stage metadata and SHOW CREATE now pass the generic credential envelope to the same codec. The SPI exchanges only JDK maps and byte arrays. GCS uses host-provided generated fe-grpc/fe-thrift types, with no dependency on fe-core. Existing Thrift/protobuf definitions and field-presence semantics are unchanged. ### Release note Filesystem plugins must be rebuilt for filesystem plugin API 3.0. Native GCP, HMAC and anonymous authentication behavior is unchanged. ### Check List (For Author) - Test: All 49 filesystem SPI tests passed, including the regenerated surface baseline and default-provider compatibility test. - Test: Standalone validation compiled current GCS/S3 provider sources, ran all 5 new credential codec test methods and 19 existing GCS provider test methods against extracted credential schemas and cached libraries. Both libthrift 0.16.0 and 0.24.0 runtimes passed; protocol sources were generated with the available thrift 0.16.0 and protoc 3.21.11 tools. - Test: Added core registry/transport coverage for plugin-defined fields and missing/ambiguous codecs. These core tests were not executed. - Test: SPI/GCS/core Checkstyle, connector import and metadata-funnel checks, dependency boundary checks, and git diff --check passed. - Test: Standard GCS reactor tests stopped before execution because the local thrift compiler is 0.16.0 while master requires 0.24.0. Full FE tests and packaged-plugin integration tests were not run. - Behavior changed: Implementation ownership and filesystem plugin API. - Does this need documentation: Filesystem developer README updated. - Self-review: Provider-specific conversion is confined to GCS; generic core paths handle CREATE, partial ALTER, BE transport, Cloud stages and SHOW CREATE. Codec state is immutable/stateless, serializers are per-call, and conversion adds no network I/O, threads, caches or owned resources. Missing/duplicate codecs and malformed envelopes fail explicitly. Partial ALTER preserves omitted versus explicitly cleared fields. No protocol ids, persistence formats or data visibility paths changed. Existing masking remains in SHOW CREATE; no credential logging is added. --- .../doris/cloud/storage/ObjectInfo.java | 15 +-- .../cloud/storage/ObjectInfoAdapter.java | 12 +- .../property/storage/auth/ObjCredential.java | 34 ----- .../storage/auth/ObjCredentialFactory.java | 46 ------- .../storage/CloudObjectStoreAdapter.java | 6 +- .../ObjectStorageCredentialAdapter.java | 88 +++++++++++++ .../datasource/storage/S3ThriftAdapter.java | 3 +- .../datasource/storage/StorageAdapter.java | 20 ++- .../ShowCreateStorageVaultCommand.java | 7 +- .../fs/ObjectStorageCredentialCodecTest.java | 124 ++++++++++++++++++ fe/fe-filesystem/README.md | 14 ++ fe/fe-filesystem/fe-filesystem-gcs/pom.xml | 14 ++ .../filesystem/gcs/GcsCredentialCodec.java} | 67 +++++----- .../filesystem/gcs/GcsFileSystemProvider.java | 6 + .../gcs/GcsCredentialCodecTest.java | 113 ++++++++++++++++ .../filesystem/spi/FileSystemProvider.java | 5 + .../spi/ObjectStorageCredentialCodec.java | 42 ++++++ .../spi/FileSystemPluginSurfaceTest.java | 1 + ...FileSystemProviderTypedPropertiesTest.java | 5 + .../resources/filesystem-plugin-surface.txt | 5 + fe/fe-filesystem/pom.xml | 2 +- 21 files changed, 494 insertions(+), 135 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java rename fe/{fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java => fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java} (60%) create mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java create mode 100644 fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java index 7f9d84ec743722..1494810c4d147a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java +++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfo.java @@ -38,7 +38,7 @@ public class ObjectInfo { // only used for aws private final String externalId; private final String token; - private final Cloud.GcpCredentialPB gcpCredential; + private final Cloud.ObjectStoreCredentialPB credential; // Used to get sts token public ObjectInfo(Cloud.ObjectStoreInfoPB.Provider provider, String ak, String sk, @@ -75,8 +75,8 @@ public ObjectInfo(Cloud.ObjectStoreInfoPB.Provider provider, String ak, String s private ObjectInfo(Cloud.ObjectStoreInfoPB.Provider provider, String ak, String sk, String bucket, String endpoint, String region, String prefix, String roleName, String arn, String externalId, - String token, Cloud.GcpCredentialPB gcpCredential) { - this.gcpCredential = gcpCredential; + String token, Cloud.ObjectStoreCredentialPB credential) { + this.credential = credential; this.provider = provider; this.ak = ak; this.sk = sk; @@ -94,13 +94,12 @@ public Cloud.ObjectStoreInfoPB.Provider getProvider() { return provider; } - private static Cloud.GcpCredentialPB nativeCredential(Cloud.ObjectStoreInfoPB info) { - return info.hasCredential() && info.getCredential().hasGcpCredential() - ? info.getCredential().getGcpCredential() : null; + private static Cloud.ObjectStoreCredentialPB nativeCredential(Cloud.ObjectStoreInfoPB info) { + return info.hasCredential() ? info.getCredential() : null; } - public Cloud.GcpCredentialPB getGcpCredential() { - return gcpCredential; + public Cloud.ObjectStoreCredentialPB getCredential() { + return credential; } public String getAk() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java index 5b96eebbbd047f..c1fa1ecaed6fde 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/storage/ObjectInfoAdapter.java @@ -22,7 +22,7 @@ import org.apache.doris.cloud.proto.Cloud.StagePB.StageAccessType; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Config; -import org.apache.doris.datasource.property.storage.auth.GcpCredentialAdapter; +import org.apache.doris.datasource.storage.ObjectStorageCredentialAdapter; import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.filesystem.spi.StsCredentials; @@ -71,6 +71,13 @@ private ObjectInfoAdapter() {} * it separately to {@code listObjectsWithPrefix} / {@code headObjectWithMeta}. */ public static StorageAdapter toStorageAdapter(ObjectInfo obj) { + if (obj.getCredential() != null) { + Map properties = buildS3CompatibleProps(obj); + properties.put("provider", obj.getProvider().name()); + properties.putAll(ObjectStorageCredentialAdapter.toProperties( + obj.getProvider().name(), obj.getCredential())); + return StorageAdapter.of(properties); + } switch (obj.getProvider()) { case OSS: return StorageAdapter.ofProvider("OSS", buildS3CompatibleProps(obj)); @@ -79,9 +86,6 @@ public static StorageAdapter toStorageAdapter(ObjectInfo obj) { case GCP: Map gcpProperties = buildS3CompatibleProps(obj); gcpProperties.put("provider", "GCP"); - if (obj.getGcpCredential() != null) { - gcpProperties.putAll(GcpCredentialAdapter.toProperties(obj.getGcpCredential())); - } return StorageAdapter.ofProvider("GCS", gcpProperties); case COS: return StorageAdapter.ofProvider("COS", buildS3CompatibleProps(obj)); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java deleted file mode 100644 index 2f6c9794f92eb9..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredential.java +++ /dev/null @@ -1,34 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.property.storage.auth; - -import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; -import org.apache.doris.thrift.TS3StorageParam; - -/** - * Provider-native credential that can be serialized for Cloud or BE. - * - *

This transport abstraction is deliberately independent of {@code S3ObjStorage}. - * Runtime S3 client creation is extended through - * {@code AbstractS3CompatibleProperties.createS3Client} instead. - */ -public interface ObjCredential { - void applyTo(ObjectStoreInfoPB.Builder builder); - - void applyTo(TS3StorageParam param); -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java deleted file mode 100644 index 82f68aa4566661..00000000000000 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/ObjCredentialFactory.java +++ /dev/null @@ -1,46 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.datasource.property.storage.auth; - -import org.apache.doris.datasource.storage.StorageAdapter; -import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; - -import java.util.Map; -import java.util.Optional; - -/** Converts plugin-resolved native credentials to the core wire protocols. */ -public final class ObjCredentialFactory { - private ObjCredentialFactory() { - } - - public static Optional fromProperties(Map properties) { - return fromAuthentication(StorageAdapter.resolveAuthentication(properties)); - } - - public static Optional fromAuthentication(Optional authentication) { - return authentication.filter(ObjectStorageAuthentication::isNative) - .map(auth -> fromCredential(auth.getProvider(), auth.getCredential())); - } - - public static ObjCredential fromCredential(String provider, Map credential) { - if ("GCP".equals(provider)) { - return new GcpCredentialAdapter(credential); - } - throw new IllegalArgumentException("Unsupported native credential wire format: " + provider); - } -} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java index 69166f27a81e13..ee356faaf28052 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java @@ -21,7 +21,6 @@ import org.apache.doris.cloud.proto.Cloud.CredProviderTypePB; import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB.Provider; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; -import org.apache.doris.datasource.property.storage.auth.ObjCredentialFactory; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; @@ -122,7 +121,7 @@ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map credential.applyTo(builder)); return builder; } @@ -132,7 +131,8 @@ public static void applyCredentialPatch(Cloud.ObjectStoreInfoPB.Builder builder, builder.clearCredential(); StorageAdapter.resolveAuthentication(properties) .filter(auth -> !auth.getCredentialUpdates().isEmpty()) - .ifPresent(auth -> ObjCredentialFactory.fromCredential(auth.getProvider(), auth.getCredentialUpdates()) + .ifPresent(auth -> ObjectStorageCredentialAdapter.fromCredential( + auth.getProvider(), auth.getCredentialUpdates()) .applyTo(builder)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java new file mode 100644 index 00000000000000..097e2b84b2764d --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java @@ -0,0 +1,88 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.storage; + +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; +import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; +import org.apache.doris.thrift.TCredential; +import org.apache.doris.thrift.TObjStorageType; +import org.apache.doris.thrift.TS3StorageParam; + +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TException; +import org.apache.thrift.protocol.TBinaryProtocol; + +import java.io.IOException; +import java.util.Map; +import java.util.Optional; + +/** Transfers plugin-encoded credentials into core wire envelopes without interpreting provider fields. */ +public final class ObjectStorageCredentialAdapter { + private final ObjectStorageCredentialCodec codec; + private final Map fields; + + private ObjectStorageCredentialAdapter(ObjectStorageCredentialCodec codec, Map fields) { + this.codec = codec; + this.fields = Map.copyOf(fields); + } + + public static Optional fromAuthentication( + Optional authentication) { + return authentication.filter(ObjectStorageAuthentication::isNative) + .map(auth -> fromCredential(auth.getProvider(), auth.getCredential())); + } + + public static ObjectStorageCredentialAdapter fromCredential(String provider, Map fields) { + return new ObjectStorageCredentialAdapter(StorageAdapter.credentialCodec(provider), fields); + } + + public static Map toProperties(String provider, ObjectStoreCredentialPB credential) { + try { + return StorageAdapter.credentialCodec(provider).toProperties(credential.toByteArray()); + } catch (IOException e) { + throw new IllegalArgumentException("Cannot decode native object-storage credentials", e); + } + } + + public void applyTo(ObjectStoreInfoPB.Builder builder) { + try { + ObjectStoreCredentialPB credential = ObjectStoreCredentialPB.parseFrom(codec.toCloudCredential(fields)); + if (!builder.hasProvider()) { + builder.setProvider(ObjectStoreInfoPB.Provider.valueOf(codec.provider())); + } + builder.setCredential(credential); + } catch (IOException e) { + throw new IllegalArgumentException("Cannot encode native object-storage credentials for Cloud", e); + } + } + + public void applyTo(TS3StorageParam param) { + try { + TCredential credential = new TCredential(); + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(credential, codec.toThriftCredential(fields)); + if (!param.isSetProvider()) { + param.setProvider(TObjStorageType.valueOf(codec.provider())); + } + param.setCredential(credential); + } catch (IOException | TException e) { + throw new IllegalArgumentException("Cannot encode native object-storage credentials for BE", e); + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java index 843b3b5e92c152..bbbb3478946473 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java @@ -18,7 +18,6 @@ package org.apache.doris.datasource.storage; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; -import org.apache.doris.datasource.property.storage.auth.ObjCredentialFactory; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.thrift.TCredProviderType; @@ -106,7 +105,7 @@ public static TS3StorageParam getS3TStorageParam(Map properties) ? DEFAULT_CONNECTION_TIMEOUT_MS : connTimeoutMs)); String usePathStyle = properties.getOrDefault(USE_PATH_STYLE, "false"); s3Info.setUsePathStyle(Boolean.parseBoolean(usePathStyle)); - ObjCredentialFactory.fromAuthentication(authentication) + ObjectStorageCredentialAdapter.fromAuthentication(authentication) .ifPresent(credential -> credential.applyTo(s3Info)); if (authentication.filter(ObjectStorageAuthentication::isAnonymous).isPresent() || (authentication.isEmpty() && getCredentialsProviderMode(properties, diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index 35b547960b8a7a..fc54eff667af31 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -25,6 +25,7 @@ import org.apache.doris.filesystem.properties.HadoopStorageProperties; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.filesystem.spi.FileSystemProvider; +import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; import org.apache.doris.foundation.property.StoragePropertiesException; import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.fs.FileSystemPluginManager; @@ -176,13 +177,24 @@ public static boolean isClearableProperty(String key) { return manager().getProviders().stream().anyMatch(provider -> provider.clearablePropertyKeys().contains(key)); } - public static Map credentialToProperties(String providerName, Map credential) { + /** Resolve a native wire codec through the registered filesystem providers. */ + public static ObjectStorageCredentialCodec credentialCodec(String providerName) { + ObjectStorageCredentialCodec selected = null; for (FileSystemProvider provider : manager().getProviders()) { - if (provider.name().equalsIgnoreCase(providerName)) { - return provider.credentialToProperties(credential); + Optional codec = provider.credentialCodec(); + if (codec.isPresent() && codec.get().provider().equalsIgnoreCase(providerName)) { + if (selected != null) { + throw new StoragePropertiesException( + "Multiple credential codecs for provider '" + providerName + "'"); + } + selected = codec.get(); } } - throw new StoragePropertiesException("Filesystem provider '" + providerName + "' is not available"); + if (selected == null) { + throw new StoragePropertiesException("Native credential codec for provider '" + providerName + + "' is not available"); + } + return selected; } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java index 68072e070a61fd..7645ea5802912f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateStorageVaultCommand.java @@ -28,7 +28,7 @@ import org.apache.doris.common.Config; import org.apache.doris.common.FeConstants; import org.apache.doris.common.util.DatasourcePrintableMap; -import org.apache.doris.datasource.property.storage.auth.GcpCredentialAdapter; +import org.apache.doris.datasource.storage.ObjectStorageCredentialAdapter; import org.apache.doris.mysql.privilege.AccessControllerManager; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.PlanType; @@ -131,8 +131,9 @@ private String getObjectCreateStmt(Cloud.ObjectStoreInfoPB objectInfo) { properties.put("s3.region", objectInfo.getRegion()); properties.put("s3.root.path", objectInfo.getPrefix()); properties.put("s3.bucket", objectInfo.getBucket()); - if (objectInfo.hasCredential() && objectInfo.getCredential().hasGcpCredential()) { - properties.putAll(GcpCredentialAdapter.toProperties(objectInfo.getCredential().getGcpCredential())); + if (objectInfo.hasCredential()) { + properties.putAll(ObjectStorageCredentialAdapter.toProperties( + objectInfo.getProvider().name(), objectInfo.getCredential())); } else { // Masking even empty keys would produce nonempty HMAC credentials conflicting with native auth. properties.put("s3.access_key", objectInfo.getAk()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java new file mode 100644 index 00000000000000..ae4e50972754a0 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java @@ -0,0 +1,124 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.fs; + +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.datasource.storage.ObjectStorageCredentialAdapter; +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.properties.FileSystemProperties; +import org.apache.doris.filesystem.spi.FileSystemProvider; +import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; +import org.apache.doris.foundation.property.StoragePropertiesException; + +import com.google.protobuf.UnknownFieldSet; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Map; +import java.util.Optional; + +class ObjectStorageCredentialCodecTest { + @AfterEach + void restoreBuiltinRegistry() { + StorageAdapter.initPluginManager(null); + } + + @Test + void registeredPluginOwnsTheWireFieldsAndPropertyMapping() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + RecordingCodec codec = new RecordingCodec(); + manager.registerProvider(provider("external-codec-plugin", codec)); + StorageAdapter.initPluginManager(manager); + Assertions.assertSame(codec, StorageAdapter.credentialCodec("oss")); + Map fields = Map.of("plugin-owned-field", "value"); + ObjectStoreInfoPB.Builder builder = ObjectStoreInfoPB.newBuilder(); + ObjectStorageCredentialAdapter.fromCredential("OSS", fields).applyTo(builder); + Assertions.assertEquals(ObjectStoreInfoPB.Provider.OSS, builder.getProvider()); + Assertions.assertEquals(fields, codec.fields); + Assertions.assertEquals(codec.envelope, builder.getCredential()); + Assertions.assertEquals(Map.of("plugin.property", "restored"), + ObjectStorageCredentialAdapter.toProperties("OSS", builder.getCredential())); + } + + @Test + void missingOrAmbiguousCodecFailsWithoutFallingBackToAnotherProvider() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + StorageAdapter.initPluginManager(manager); + Assertions.assertThrows(StoragePropertiesException.class, () -> StorageAdapter.credentialCodec("OSS")); + manager.registerProvider(provider("first", new RecordingCodec())); + manager.registerProvider(provider("second", new RecordingCodec())); + Assertions.assertThrows(StoragePropertiesException.class, () -> StorageAdapter.credentialCodec("OSS")); + } + + private FileSystemProvider provider(String name, ObjectStorageCredentialCodec codec) { + return new FileSystemProvider() { + @Override + public String name() { + return name; + } + + @Override + public boolean supports(Map properties) { + return false; + } + + @Override + public FileSystem create(Map properties) { + throw new UnsupportedOperationException("No filesystem I/O in credential conversion"); + } + + @Override + public Optional credentialCodec() { + return Optional.of(codec); + } + }; + } + + private static class RecordingCodec implements ObjectStorageCredentialCodec { + // The kernel must transfer even fields it does not interpret. + private final ObjectStoreCredentialPB envelope = ObjectStoreCredentialPB.newBuilder() + .setUnknownFields(UnknownFieldSet.newBuilder().addField(100, + UnknownFieldSet.Field.newBuilder().addVarint(42).build()).build()).build(); + private Map fields; + + @Override + public String provider() { + return "OSS"; + } + + @Override + public byte[] toCloudCredential(Map credential) { + fields = credential; + return envelope.toByteArray(); + } + + @Override + public byte[] toThriftCredential(Map credential) { + throw new UnsupportedOperationException("This fixture only supplies Cloud credentials"); + } + + @Override + public Map toProperties(byte[] cloudCredential) { + Assertions.assertArrayEquals(envelope.toByteArray(), cloudCredential); + return Map.of("plugin.property", "restored"); + } + } +} diff --git a/fe/fe-filesystem/README.md b/fe/fe-filesystem/README.md index eb8e34ff06631a..d7ceaa141f69e8 100644 --- a/fe/fe-filesystem/README.md +++ b/fe/fe-filesystem/README.md @@ -141,6 +141,20 @@ At runtime, `DirectoryPluginRuntimeManager` creates an isolated `ClassLoader` pe `software.amazon.awssdk.*`, `org.apache.hadoop.*`) are loaded parent-first to avoid cross- ClassLoader cast failures. +### Native object-storage credential codecs + +Providers with native credentials expose an optional `ObjectStorageCredentialCodec` through +`FileSystemProvider.credentialCodec()`. The plugin owns provider-specific field names and converts +resolved credentials to the existing Cloud protobuf and BE Thrift envelopes. Only JDK maps and byte +arrays cross this SPI; the API/SPI modules retain their dependency-free protocol boundary. +`fe-core` selects the registered codec and transfers the envelopes without inspecting provider fields. +The same codec restores properties for SHOW CREATE and Cloud stages. Partial ALTER maps must retain +the distinction between an absent field and an explicitly empty value. + +The GCS codec uses host-provided `fe-grpc` and `fe-thrift` generated types (provided dependencies), +without depending on `fe-core`. Filesystem plugin API 3.0 includes this extension point; plugins must +be rebuilt for the matching host API. + ### S3 Delegation Pattern Several cloud providers (OSS, COS, OBS) are S3-compatible. Instead of duplicating the S3 diff --git a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml index d798c46d9ca285..50635596637040 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml @@ -36,6 +36,20 @@ under the License. client in fe-filesystem-s3-base. + + + org.apache.doris + fe-grpc + ${revision} + provided + + + org.apache.doris + fe-thrift + ${revision} + provided + com.google.auth google-auth-library-oauth2-http diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java similarity index 60% rename from fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java rename to fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java index bcdc0f59990d1d..a988e9c1b1bda7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/auth/GcpCredentialAdapter.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java @@ -15,42 +15,32 @@ // specific language governing permissions and limitations // under the License. -package org.apache.doris.datasource.property.storage.auth; +package org.apache.doris.filesystem.gcs; import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; -import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; -import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; import org.apache.doris.thrift.TCredential; import org.apache.doris.thrift.TGcpCredential; import org.apache.doris.thrift.TGcpCredentialProviderType; -import org.apache.doris.thrift.TObjStorageType; -import org.apache.doris.thrift.TS3StorageParam; +import org.apache.thrift.TException; +import org.apache.thrift.TSerializer; +import org.apache.thrift.protocol.TBinaryProtocol; + +import java.io.IOException; import java.util.HashMap; import java.util.Map; -/** GCP native credential shared by the Cloud protobuf and BE Thrift paths. */ -public final class GcpCredentialAdapter implements ObjCredential { - private final Map fields; - - public GcpCredentialAdapter(Map fields) { - this.fields = Map.copyOf(fields); - } - - public static Map toProperties(GcpCredentialPB credential) { - Map fields = new HashMap<>(); - fields.put("credential_provider_type", credential.getCredentialProviderType().name()); - if (credential.hasImpersonationServiceAccount()) { - fields.put("impersonation_service_account", credential.getImpersonationServiceAccount()); - } - return StorageAdapter.credentialToProperties("GCS", fields); +/** Native GCP wire encoding belongs to the GCS plugin, alongside its authentication rules. */ +public final class GcsCredentialCodec implements ObjectStorageCredentialCodec { + @Override + public String provider() { + return "GCP"; } @Override - public void applyTo(ObjectStoreInfoPB.Builder builder) { - if (!builder.hasProvider()) { - builder.setProvider(ObjectStoreInfoPB.Provider.GCP); - } + public byte[] toCloudCredential(Map fields) { GcpCredentialPB.Builder credential = GcpCredentialPB.newBuilder(); if (fields.containsKey("credential_provider_type")) { credential.setCredentialProviderType(GcpCredentialPB.CredentialProviderType.valueOf( @@ -59,14 +49,11 @@ public void applyTo(ObjectStoreInfoPB.Builder builder) { if (fields.containsKey("impersonation_service_account")) { credential.setImpersonationServiceAccount(fields.get("impersonation_service_account")); } - builder.getCredentialBuilder().setGcpCredential(credential); + return ObjectStoreCredentialPB.newBuilder().setGcpCredential(credential).build().toByteArray(); } @Override - public void applyTo(TS3StorageParam param) { - if (!param.isSetProvider()) { - param.setProvider(TObjStorageType.GCP); - } + public byte[] toThriftCredential(Map fields) throws IOException { TGcpCredential credential = new TGcpCredential(); if (fields.containsKey("credential_provider_type")) { credential.setCredentialProviderType(TGcpCredentialProviderType.valueOf( @@ -75,6 +62,26 @@ public void applyTo(TS3StorageParam param) { if (fields.containsKey("impersonation_service_account")) { credential.setImpersonationServiceAccount(fields.get("impersonation_service_account")); } - param.setCredential(new TCredential().setGcpCredential(credential)); + try { + return new TSerializer(new TBinaryProtocol.Factory()) + .serialize(new TCredential().setGcpCredential(credential)); + } catch (TException e) { + throw new IOException("Cannot encode GCS credentials for BE", e); + } + } + + @Override + public Map toProperties(byte[] cloudCredential) throws IOException { + ObjectStoreCredentialPB envelope = ObjectStoreCredentialPB.parseFrom(cloudCredential); + if (!envelope.hasGcpCredential()) { + throw new IOException("Missing GCP credentials in the Cloud credential envelope"); + } + GcpCredentialPB credential = envelope.getGcpCredential(); + Map fields = new HashMap<>(); + fields.put("credential_provider_type", credential.getCredentialProviderType().name()); + if (credential.hasImpersonationServiceAccount()) { + fields.put("impersonation_service_account", credential.getImpersonationServiceAccount()); + } + return new GcsFileSystemProvider().credentialToProperties(fields); } } diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index 00fe30bb105a5f..53392aef11426c 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -25,6 +25,7 @@ import org.apache.doris.filesystem.s3.S3FileSystem; import org.apache.doris.filesystem.s3.S3FileSystemProperties; import org.apache.doris.filesystem.spi.FileSystemProvider; +import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import java.io.IOException; @@ -107,6 +108,11 @@ public Optional resolveAuthentication(Map credentialCodec() { + return Optional.of(new GcsCredentialCodec()); + } + @Override public Map credentialToProperties(Map credential) { Map properties = new HashMap<>(); diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java new file mode 100644 index 00000000000000..d2a6efd89ab72b --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java @@ -0,0 +1,113 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.gcs; + +import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; +import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; +import org.apache.doris.thrift.TCredential; +import org.apache.doris.thrift.TGcpCredential; +import org.apache.doris.thrift.TGcpCredentialProviderType; + +import org.apache.thrift.TDeserializer; +import org.apache.thrift.protocol.TBinaryProtocol; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.util.Map; + +class GcsCredentialCodecTest { + private final GcsFileSystemProvider provider = new GcsFileSystemProvider(); + private final ObjectStorageCredentialCodec codec = provider.credentialCodec().orElseThrow(); + + @Test + void nativeCredentialsRoundTripThroughBothWireFormats() throws Exception { + for (String source : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { + Map properties = Map.of( + "provider", "GCP", + "gs.credential_provider_type", source, + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"); + ObjectStorageAuthentication auth = provider.resolveAuthentication(properties).orElseThrow(); + Assertions.assertEquals(auth.getProvider(), codec.provider()); + byte[] cloudBytes = codec.toCloudCredential(auth.getCredential()); + GcpCredentialPB cloud = ObjectStoreCredentialPB.parseFrom(cloudBytes).getGcpCredential(); + TGcpCredential thrift = decodeThrift(auth.getCredential()); + Assertions.assertEquals(GcpCredentialPB.CredentialProviderType.valueOf(source), + cloud.getCredentialProviderType()); + Assertions.assertEquals(TGcpCredentialProviderType.valueOf(source), thrift.getCredentialProviderType()); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", cloud.getImpersonationServiceAccount()); + Assertions.assertEquals(cloud.getImpersonationServiceAccount(), thrift.getImpersonationServiceAccount()); + Assertions.assertEquals(Map.of("gs.credential_provider_type", source, + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"), + codec.toProperties(cloudBytes)); + } + } + + @Test + void partialAlterPreservesExplicitClearWithoutAddingProviderType() throws Exception { + ObjectStorageAuthentication auth = provider.resolveAuthentication(Map.of( + "provider", "GCP", "gs.impersonation_service_account", "")).orElseThrow(); + GcpCredentialPB cloud = ObjectStoreCredentialPB.parseFrom( + codec.toCloudCredential(auth.getCredentialUpdates())).getGcpCredential(); + TGcpCredential thrift = decodeThrift(auth.getCredentialUpdates()); + Assertions.assertFalse(cloud.hasCredentialProviderType()); + Assertions.assertFalse(thrift.isSetCredentialProviderType()); + Assertions.assertTrue(cloud.hasImpersonationServiceAccount()); + Assertions.assertTrue(thrift.isSetImpersonationServiceAccount()); + Assertions.assertEquals("", cloud.getImpersonationServiceAccount()); + Assertions.assertEquals("", thrift.getImpersonationServiceAccount()); + } + + @Test + void omittedImpersonationStaysAbsent() throws Exception { + Map fields = Map.of("credential_provider_type", "DEFAULT"); + byte[] cloudBytes = codec.toCloudCredential(fields); + Assertions.assertFalse(ObjectStoreCredentialPB.parseFrom(cloudBytes) + .getGcpCredential().hasImpersonationServiceAccount()); + Assertions.assertFalse(decodeThrift(fields).isSetImpersonationServiceAccount()); + Assertions.assertEquals(Map.of("gs.credential_provider_type", "DEFAULT"), codec.toProperties(cloudBytes)); + } + + @Test + void invalidEnvelopeFailsInsteadOfSelectingDefaultCredentials() { + Assertions.assertThrows(IOException.class, + () -> codec.toProperties(ObjectStoreCredentialPB.getDefaultInstance().toByteArray())); + Assertions.assertThrows(IOException.class, () -> codec.toProperties(new byte[] {(byte) 0xff})); + Assertions.assertThrows(IllegalArgumentException.class, + () -> codec.toCloudCredential(Map.of("credential_provider_type", "UNKNOWN"))); + Assertions.assertThrows(IllegalArgumentException.class, + () -> codec.toThriftCredential(Map.of("credential_provider_type", "UNKNOWN"))); + } + + @Test + void hmacAndAnonymousAuthenticationDoNotProduceNativeCredentials() { + Assertions.assertFalse(provider.resolveAuthentication(Map.of("provider", "GCP", + "s3.access_key", "ak", "s3.secret_key", "sk")).orElseThrow().isNative()); + Assertions.assertFalse(provider.resolveAuthentication(Map.of("provider", "GCP", + "gs.credential_provider_type", "ANONYMOUS")).orElseThrow().isNative()); + } + + private TGcpCredential decodeThrift(Map fields) throws Exception { + TCredential envelope = new TCredential(); + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(envelope, codec.toThriftCredential(fields)); + Assertions.assertTrue(envelope.isSetGcpCredential()); + return envelope.getGcpCredential(); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java index d4bf8e374fa7e0..205d7c58a4f31b 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java @@ -61,6 +61,11 @@ default Optional resolveAuthentication(Map credentialCodec() { + return Optional.empty(); + } + /** Translate protocol-native fields back into provider-owned user properties. */ default Map credentialToProperties(Map credential) { throw new UnsupportedOperationException(name() + " has no native credential property codec"); diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java new file mode 100644 index 00000000000000..1b14f2258298dc --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java @@ -0,0 +1,42 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.spi; + +import java.io.IOException; +import java.util.Map; + +/** + * Provider-owned conversion between resolved credentials and the existing BE/Cloud wire envelopes. + * + *

Only JDK values cross the plugin boundary. Generated protocol classes and their runtimes remain + * implementation details; plugins must not depend on fe-core. Implementations perform no I/O and + * preserve absent fields versus explicitly empty fields in partial ALTER credentials.

+ */ +public interface ObjectStorageCredentialCodec { + /** Canonical object-store provider name used by the BE and Cloud protocols. */ + String provider(); + + /** Encode a complete ObjectStoreCredentialPB envelope, not its provider-specific submessage. */ + byte[] toCloudCredential(Map credential) throws IOException; + + /** Encode a complete TCredential envelope using Thrift's binary protocol. */ + byte[] toThriftCredential(Map credential) throws IOException; + + /** Restore user-facing properties from an ObjectStoreCredentialPB envelope. */ + Map toProperties(byte[] cloudCredential) throws IOException; +} diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java index 5e44a0866db6f2..dd6644e9f7890e 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java @@ -68,6 +68,7 @@ public class FileSystemPluginSurfaceTest { /** The types a filesystem plugin implements or calls. Everything reachable on them is the contract. */ private static final List> FROZEN_TYPES = Arrays.asList( FileSystemProvider.class, + ObjectStorageCredentialCodec.class, org.apache.doris.filesystem.auth.ObjectStorageAuthentication.class, ObjFileSystem.class, ObjStorage.class, diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java index 4d614f22bd3396..b1aa754e002350 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java @@ -46,6 +46,11 @@ void provider_canBindAndCreateWithTypedProperties() throws IOException { Assertions.assertSame(properties, ((TestFileSystem) fileSystem).properties); } + @Test + void providerWithoutNativeCredentialsDoesNotAdvertiseACodec() { + Assertions.assertTrue(new TypedProvider().credentialCodec().isEmpty()); + } + private static class TypedProvider implements FileSystemProvider { @Override public boolean supports(Map properties) { diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt index f88e7f19ea852a..ada0edd5d04ac4 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt @@ -18,6 +18,7 @@ org.apache.doris.filesystem.spi.FileSystemProvider#create(java.util.Map):org.apa org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.extension.spi.PluginContext):org.apache.doris.extension.spi.Plugin org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem org.apache.doris.filesystem.spi.FileSystemProvider#createUntyped(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem +org.apache.doris.filesystem.spi.FileSystemProvider#credentialCodec():java.util.Optional org.apache.doris.filesystem.spi.FileSystemProvider#credentialToProperties(java.util.Map):java.util.Map org.apache.doris.filesystem.spi.FileSystemProvider#description():java.lang.String org.apache.doris.filesystem.spi.FileSystemProvider#modifiableCredentialPropertyKeys():java.util.Set @@ -72,3 +73,7 @@ org.apache.doris.filesystem.spi.ObjStorage#listObjectsWithPrefix(java.lang.Strin org.apache.doris.filesystem.spi.ObjStorage#openInputStreamAt(java.lang.String,long):java.io.InputStream org.apache.doris.filesystem.spi.ObjStorage#putObject(java.lang.String,org.apache.doris.filesystem.spi.RequestBody):void org.apache.doris.filesystem.spi.ObjStorage#uploadPart(java.lang.String,java.lang.String,int,org.apache.doris.filesystem.spi.RequestBody):org.apache.doris.filesystem.UploadPartResult +org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#provider():java.lang.String +org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#toCloudCredential(java.util.Map):byte[] +org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#toProperties(byte[]):java.util.Map +org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#toThriftCredential(java.util.Map):byte[] diff --git a/fe/fe-filesystem/pom.xml b/fe/fe-filesystem/pom.xml index 73a8e6bf4a96f5..d8a6e8609463da 100644 --- a/fe/fe-filesystem/pom.xml +++ b/fe/fe-filesystem/pom.xml @@ -53,7 +53,7 @@ under the License. bumping all five families. See plan-doc/designs/2026-07-29-plugin-api-version-check-design.md. --> - 2.0 + 3.0 From c7d9d04bc4d9ccf9a4f3a71a9681810147819304 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 12 Oct 2026 15:37:22 +0800 Subject: [PATCH 30/47] [refactor](fe) Keep GCS routing and credential rules out of S3 plugins ### What problem does this PR solve? Related PR: #68540 Problem Summary: GCS native authentication previously changed S3 provider selection and validation and made the shared S3 protocol library depend on GCS authentication classes. Restore the S3, S3-base and MinIO modules to upstream and move the new GCS detection and credential checks into the GCS provider. The filesystem registry now validates plugin-owned properties before selection and lets matched plugins declare general heuristic fallbacks. Explicit provider selection wins, bindPrimary and bindAll agree, and catalog HDFS fallback remains available. Forced S3 binding also rejects native GCP credentials. Declare the GCS-common dependency directly in GCS and Iceberg, which actually use it. Bump the filesystem plugin API to 4.0 and regenerate its surface baseline for the two default hooks. GCS still reuses the existing S3-base protocol client; it does not depend on the S3 plugin. ### Release note None ### Check List (For Author) - Test: Maven reactor suites for filesystem SPI, GCS-common, S3-base, S3, MinIO and extension-loader passed; Checkstyle and connector import/metadata gates passed. An independent harness compiled the actual manager, storage adapter and plugin sources and passed all 14 routing tests, plus 25 GCS provider/codec tests. The harness substitutes host configuration/masking and unused URI helpers and generates isolated credential schemas; it is not the full FE suite. Standard FE UT is blocked by the local JDK configuration; the normal GCS reactor is blocked by thrift compiler 0.16 versus required 0.24. - Behavior changed: Yes, explicit registered provider names take precedence over heuristics; GCS-owned signals supersede S3/MinIO fallbacks without changing those plugins. - Does this need documentation: Yes, filesystem README updated in this commit. Self-review: Scope is plugin selection, credential validation and dependencies. No transaction, persistence, data-write, FE/BE field, configuration, lock, thread or client-lifecycle changes. Selection remains deterministic and performs no I/O; providers retain the existing CopyOnWriteArrayList lifecycle. Invalid credentials fail before selecting another provider. SPI compatibility is handled by the version/baseline bump; no protocol schemas change. Primary, multi-binding, raw creation and forced binding paths are covered. Positive/negative routing and unchanged S3/MinIO suites pass, subject to the full-build limitations above. Existing exceptions identify invalid selection; no additional metrics or logging are needed for these local rules. --- fe/fe-connector/fe-connector-iceberg/pom.xml | 5 + .../datasource/storage/StorageAdapter.java | 3 + .../doris/fs/FileSystemPluginManager.java | 83 ++++++++++--- .../doris/fs/FileSystemPluginManagerTest.java | 110 ++++++++++++++++++ .../apache/doris/fs/GcsGuessRoutingTest.java | 31 +++++ fe/fe-filesystem/README.md | 18 ++- .../filesystem/gcs/auth/GcsAuthResolver.java | 3 +- fe/fe-filesystem/fe-filesystem-gcs/pom.xml | 5 + .../filesystem/gcs/GcsFileSystemProvider.java | 20 +++- .../gcs/GcsFileSystemProviderTest.java | 14 +++ .../minio/MinioFileSystemProvider.java | 4 - .../minio/MinioFileSystemProviderTest.java | 12 -- .../fe-filesystem-s3-base/pom.xml | 5 - .../doris/filesystem/s3/S3CompatSignals.java | 8 +- .../filesystem/s3/S3FileSystemProperties.java | 4 - .../s3/S3FileSystemPropertiesTest.java | 6 - .../filesystem/s3/S3FileSystemProvider.java | 7 +- .../s3/S3FileSystemProviderTest.java | 40 ------- .../filesystem/spi/FileSystemProvider.java | 17 +++ .../resources/filesystem-plugin-surface.txt | 2 + fe/fe-filesystem/pom.xml | 2 +- 21 files changed, 300 insertions(+), 99 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/pom.xml b/fe/fe-connector/fe-connector-iceberg/pom.xml index e474b0cf8d7f63..2109a405cca557 100644 --- a/fe/fe-connector/fe-connector-iceberg/pom.xml +++ b/fe/fe-connector/fe-connector-iceberg/pom.xml @@ -41,6 +41,11 @@ under the License. + + org.apache.doris + fe-filesystem-gcs-common + ${revision} + org.apache.doris diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index fc54eff667af31..b041a5c48ae59a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -226,6 +226,9 @@ public static List ofAll(Map origProps) { */ public static StorageAdapter ofProvider(String providerName, Map origProps) { Map props = origProps == null ? new HashMap<>() : origProps; + Map selection = new HashMap<>(props); + selection.put("provider", providerName); + manager().validateProperties(selection); for (FileSystemProvider provider : manager().getProviders()) { if (provider.name().equalsIgnoreCase(providerName)) { return new StorageAdapter(provider.bind(withHadoopConfigDir(props)), props); diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java index 4c3d59548307cf..a5564f8ffb1470 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java @@ -45,6 +45,7 @@ import java.util.Map; import java.util.ServiceLoader; import java.util.Set; +import java.util.TreeSet; import java.util.concurrent.CopyOnWriteArrayList; /** @@ -54,8 +55,9 @@ * 1. ServiceLoader scan (classpath-based built-ins / test overrides) * 2. DirectoryPluginRuntimeManager scan (production plugin directories) * - *

The first provider that returns {@code supports(props) == true} is used. - * Classpath providers have higher priority than directory-loaded providers. + *

Explicit selections precede heuristic matches, and matching providers may supersede their + * declared fallbacks. Remaining ties retain registration order. Classpath providers have higher + * priority than directory-loaded providers. */ public class FileSystemPluginManager { @@ -200,8 +202,24 @@ private boolean hasProviderNamed(String name) { * @throws IOException if no provider supports the properties, or creation fails */ public FileSystem createFileSystem(Map properties) throws IOException { + validateProperties(properties); + FileSystemProvider named = providerByName(properties.get("provider")); + if (named != null) { + return named.create(properties); + } for (FileSystemProvider provider : providers) { + if (provider.supportsExplicit(properties)) { + return provider.create(properties); + } + } + Set fallbacks = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (FileSystemProvider provider : providers) { if (provider.supports(properties)) { + fallbacks.addAll(provider.fallbackProviders()); + } + } + for (FileSystemProvider provider : providers) { + if (!fallbacks.contains(provider.name()) && provider.supports(properties)) { return provider.create(properties); } } @@ -228,32 +246,68 @@ public List getProviders() { private static final String DEPRECATED_OSS_HDFS_SUPPORT = "oss.hdfs.enabled"; + /** Validate plugin-owned keys even if routing would otherwise ignore them. */ + public void validateProperties(Map properties) { + for (FileSystemProvider provider : providers) { + provider.validateProperties(properties); + } + } + + private boolean useGuess(Map properties) { + return !hasAnyExplicitFsSupport(properties) && providerByName(properties.get("provider")) == null; + } + + private static boolean matchesExplicit(FileSystemProvider provider, Map properties) { + return provider.name().equalsIgnoreCase(properties.get("provider")) || provider.supportsExplicit(properties); + } + + private Set fallbackProviders(Map properties, Map probeView, + boolean useGuess) { + Set fallbacks = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (FileSystemProvider provider : providers) { + if (matchesExplicit(provider, properties) || (useGuess && provider.supportsGuess(probeView))) { + fallbacks.addAll(provider.fallbackProviders()); + } + } + return fallbacks; + } + + private static boolean matches(FileSystemProvider provider, Map properties, + Map probeView, boolean useGuess, Set fallbacks) { + return matchesExplicit(provider, properties) + || (useGuess && !fallbacks.contains(provider.name()) && provider.supportsGuess(probeView)); + } + /** * Binds the first matching provider for raw user properties, mirroring * {@code StorageProperties.createPrimary}: fixed priority, explicit {@code fs..support} - * flags first-class, heuristic guesses globally disabled once any explicit flag is present, + * flags first-class, heuristic guesses disabled by an explicit flag or a registered provider name, + * matched plugins superseding their declared heuristic fallbacks, * and no default fallback — throws when nothing matches. */ public FileSystemProperties bindPrimary(Map properties) { - boolean useGuess = !hasAnyExplicitFsSupport(properties); + validateProperties(properties); + boolean useGuess = useGuess(properties); Map probeView = withProbeContext(properties); + Set fallbacks = fallbackProviders(properties, probeView, useGuess); for (StorageRegistry.Provider meta : StorageRegistry.Provider.values()) { FileSystemProvider provider = providerByName(meta.name()); if (provider == null) { continue; } - if (provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView))) { + if (matches(provider, properties, probeView, useGuess, fallbacks)) { return provider.bind(properties); } } // Providers not in the StorageRegistry.Provider table (out-of-tree plugins) are consulted // after every known provider, in registration order — adding a plugin needs no fe-core - // priority-list change and can never preempt the built-in set. + // priority-list change. A matched plugin can supersede its declared fallback providers. for (FileSystemProvider provider : providers) { if (StorageRegistry.Provider.byName(provider.name()).isPresent()) { continue; } - FileSystemProperties bound = tryBindUnlisted(provider, properties, useGuess, probeView); + FileSystemProperties bound = tryBindUnlisted(provider, properties, + useGuess && !fallbacks.contains(provider.name()), probeView); if (bound != null) { return bound; } @@ -270,8 +324,10 @@ public FileSystemProperties bindPrimary(Map properties) { * default HDFS binding is inserted at index 0 (fe-core's default-HDFS fallback). */ public List bindAll(Map properties) { - boolean useGuess = !hasAnyExplicitFsSupport(properties); + validateProperties(properties); + boolean useGuess = useGuess(properties); Map probeView = withProbeContext(properties); + Set fallbacks = fallbackProviders(properties, probeView, useGuess); List result = new ArrayList<>(); boolean ossHdfsMatched = false; boolean hdfsFamilyMatched = false; @@ -291,7 +347,7 @@ public List bindAll(Map properties) { if (meta == StorageRegistry.Provider.HDFS && hdfsFamilyMatched) { continue; } - if (provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView))) { + if (matches(provider, properties, probeView, useGuess, fallbacks)) { result.add(provider.bind(properties)); if (meta == StorageRegistry.Provider.OSS_HDFS) { ossHdfsMatched = true; @@ -310,12 +366,13 @@ public List bindAll(Map properties) { if (StorageRegistry.Provider.byName(provider.name()).isPresent()) { continue; } - FileSystemProperties bound = tryBindUnlisted(provider, properties, useGuess, probeView); + FileSystemProperties bound = tryBindUnlisted(provider, properties, + useGuess && !fallbacks.contains(provider.name()), probeView); if (bound != null) { result.add(bound); } } - if (useGuess && !hdfsFamilyMatched) { + if (!hasAnyExplicitFsSupport(properties) && !hdfsFamilyMatched) { FileSystemProvider hdfs = providerByName("HDFS"); if (hdfs != null) { result.add(0, hdfs.bind(properties)); @@ -344,7 +401,7 @@ public List bindAll(Map properties) { */ private FileSystemProperties tryBindUnlisted(FileSystemProvider provider, Map properties, boolean useGuess, Map probeView) { - boolean typedMatch = provider.supportsExplicit(properties) + boolean typedMatch = matchesExplicit(provider, properties) || (useGuess && provider.supportsGuess(probeView)); if (!typedMatch) { if (useGuess && provider.supports(properties)) { @@ -379,7 +436,7 @@ public static Map withProbeContext(Map propertie private FileSystemProvider providerByName(String name) { for (FileSystemProvider p : providers) { - if (name.equalsIgnoreCase(p.name())) { + if (p.name().equalsIgnoreCase(name)) { return p; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java index 29feec06d54b2e..df9bb4b7224560 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java @@ -123,6 +123,116 @@ public void bindAll_returnsEmptyListWhenNoProviderSupports() { // modules: runtime dependencies removed in Phase 4 P4.1"). End-to-end binding against the real // providers is therefore covered by P1-T06 (docker / full plugin classpath), not here. + @Test + public void specificPluginSupersedesFallbackInPrimaryAndAllBindings() throws Exception { + for (boolean specificFirst : new boolean[] {false, true}) { + FileSystemPluginManager manager = new FileSystemPluginManager(); + FileSystemProvider specific = routingProvider("custom", Set.of("s3")); + FileSystemProvider general = routingProvider("S3", Set.of()); + manager.registerProvider(specificFirst ? specific : general); + manager.registerProvider(specificFirst ? general : specific); + Map props = Map.of("custom.match", "true", "S3.match", "true"); + Assertions.assertEquals("custom", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("custom", manager.bindAll(props).get(0).providerName()); + Assertions.assertEquals("custom", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + + // A nonmatching specific plugin must not remove the general fallback. + Assertions.assertEquals("S3", manager.bindPrimary(Map.of("S3.match", "true")).providerName()); + } + } + + @Test + public void explicitSelectionWinsAndMultipleExplicitFlagsRemainSupported() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("custom", Set.of("s3"))); + manager.registerProvider(routingProvider("S3", Set.of())); + Map props = new HashMap<>(Map.of("custom.match", "true", "S3.match", "true")); + props.put("provider", "s3"); + Assertions.assertEquals("S3", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("S3", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + props.remove("provider"); + props.put("fs.s3.support", "true"); + Assertions.assertEquals("S3", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("S3", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + props.put("fs.custom.support", "true"); + Assertions.assertEquals(2, manager.bindAll(props).size()); + } + + @Test + public void namedProviderPreservesDefaultHdfsAndExplicitFlagsDisableIt() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("S3", Set.of())); + manager.registerProvider(routingProvider("HDFS", Set.of())); + List bindings = manager.bindAll(Map.of("provider", "s3")); + Assertions.assertEquals(2, bindings.size()); + Assertions.assertEquals("HDFS", bindings.get(0).providerName()); + Assertions.assertEquals("S3", bindings.get(1).providerName()); + Assertions.assertEquals(1, manager.bindAll(Map.of("fs.s3.support", "true")).size()); + } + + @Test + public void nonSelectedPluginStillValidatesItsOwnedProperties() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("custom", Set.of())); + manager.registerProvider(routingProvider("S3", Set.of())); + Map props = Map.of("provider", "S3", "custom.invalid", "true"); + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.bindPrimary(props)); + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.bindAll(props)); + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.createFileSystem(props)); + } + + private static FileSystemProvider routingProvider(String name, Set fallbacks) { + return new FileSystemProvider() { + @Override + public Set fallbackProviders() { + return fallbacks; + } + + @Override + public void validateProperties(Map properties) { + if (properties.containsKey(name + ".invalid")) { + throw new IllegalArgumentException("Invalid " + name + " properties"); + } + } + + @Override + public boolean supportsExplicit(Map properties) { + return "true".equals(properties.get("fs." + name.toLowerCase(java.util.Locale.ROOT) + ".support")); + } + + @Override + public boolean supportsGuess(Map properties) { + return properties.containsKey(name + ".match"); + } + + @Override + public boolean supports(Map properties) { + return supportsExplicit(properties) || supportsGuess(properties); + } + + @Override + public FileSystemProperties bind(Map properties) { + return new FakeFsProps(name); + } + + @Override + public FileSystem create(Map properties) { + throw new UnsupportedOperationException(name); + } + + @Override + public String name() { + return name; + } + }; + } + // ---- helpers ---- private static FileSystemProvider bindingProvider( diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java index 4acbe26f70add8..d62bf7b7acc699 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java @@ -79,6 +79,37 @@ public void testInferredNativeGcsStillValidatesEndpoint() { } } + @Test + public void testNativeGcsHintsWithoutEndpointDoNotBindS3OrMinio() { + for (Map.Entry hint : Map.of( + "provider", "GCP", "gs.credential_provider_type", "DEFAULT", + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com", + "warehouse", "gs://bucket/warehouse").entrySet()) { + Map props = new HashMap<>(); + props.put("s3.region", "us-east1"); + props.put("_STORAGE_TYPE_", "S3"); + props.put(hint.getKey(), hint.getValue()); + bindings(props, "GCS"); + } + bindings(Map.of("s3.region", "us-east1", "gs.impersonation_service_account", ""), "GCS"); + bindings(Map.of("s3.region", "us-east-1"), "S3"); + } + + @Test + public void testNativeCredentialsCannotBeIgnoredByS3Selection() { + for (Map selection : List.of(Map.of("provider", "S3"), + Map.of("fs.s3.support", "true"))) { + Map props = new HashMap<>(selection); + props.put("gs.credential_provider_type", "DEFAULT"); + props.put("s3.region", "us-east-1"); + Assertions.assertThrows(IllegalArgumentException.class, () -> StorageAdapter.of(props)); + Assertions.assertThrows(IllegalArgumentException.class, () -> StorageAdapter.ofAll(props)); + } + Assertions.assertThrows(IllegalArgumentException.class, + () -> StorageAdapter.ofProvider("S3", Map.of("s3.region", "us-east-1", + "gs.credential_provider_type", "DEFAULT"))); + } + private static Map properties(String endpoint) { Map props = new HashMap<>(); props.put("s3.endpoint", endpoint); diff --git a/fe/fe-filesystem/README.md b/fe/fe-filesystem/README.md index d7ceaa141f69e8..127cf6e8cc029f 100644 --- a/fe/fe-filesystem/README.md +++ b/fe/fe-filesystem/README.md @@ -152,9 +152,25 @@ The same codec restores properties for SHOW CREATE and Cloud stages. Partial ALT the distinction between an absent field and an explicitly empty value. The GCS codec uses host-provided `fe-grpc` and `fe-thrift` generated types (provided dependencies), -without depending on `fe-core`. Filesystem plugin API 3.0 includes this extension point; plugins must +without depending on `fe-core`. Filesystem plugin API 4.0 includes this extension point; plugins must be rebuilt for the matching host API. +### Overlapping provider matches + +The binding registry honors an explicit registered `provider` name or `fs..support` flags before +heuristic matching. A provider can declare `fallbackProviders()` to suppress general-purpose +heuristic matches when it also matches. This applies to both primary and multi-provider binding; +explicit matches are never suppressed. Providers without this declaration retain their existing +multi-match behavior. The default HDFS binding for catalogs is preserved. + +`validateProperties()` runs on every loaded provider before selection, so invalid provider-owned +credentials cannot silently disappear when another plugin is chosen. It must be deterministic and +perform no I/O. Forced provider binding validates with the effective provider name too. + +GCS owns its endpoint/credential recognition and declares S3 and MinIO as heuristic fallbacks. +S3 and its shared protocol library do not depend on GCS authentication classes. Filesystem plugin +API 4.0 adds these two default hooks; all plugins must be rebuilt for that host API. + ### S3 Delegation Pattern Several cloud providers (OSS, COS, OBS) are S3-compatible. Instead of duplicating the S3 diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java index e0b7b34aece015..6b748b40336bb4 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java @@ -112,8 +112,7 @@ public static boolean guessIsGcs(Map properties) { if ("provider".equals(key) && ("GCP".equalsIgnoreCase(value) || "GCS".equalsIgnoreCase(value))) { return true; } - if ((CREDENTIAL_PROVIDER_TYPE.equals(key) || IMPERSONATION_SERVICE_ACCOUNT.equals(key)) - && isNotBlank(value)) { + if (CREDENTIAL_PROVIDER_TYPE.equals(key) || IMPERSONATION_SERVICE_ACCOUNT.equals(key)) { return true; } if (("uri".equals(key) || "warehouse".equals(key)) diff --git a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml index 50635596637040..4c66a21730509c 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml @@ -36,6 +36,11 @@ under the License. client in fe-filesystem-s3-base. + + org.apache.doris + fe-filesystem-gcs-common + ${revision} + diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index 53392aef11426c..b504613312fc0a 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -41,7 +41,7 @@ * *

Selected explicitly by {@code provider=GCS}/{@code GCP} or {@code fs.gcs.support=true}; * otherwise, when the user declared no filesystem explicitly, by - * {@link S3CompatSignals#guessIsGcs} (a {@code gs.endpoint}, or an endpoint on + * {@link GcsAuthResolver#guessIsGcs} (a {@code gs.endpoint}, or an endpoint on * {@code storage.googleapis.com}). Delegates I/O to {@link S3FileSystem}. */ public class GcsFileSystemProvider implements FileSystemProvider { @@ -108,6 +108,20 @@ public Optional resolveAuthentication(Map properties) { + GcsAuthResolver.resolve(properties); + if (GcsAuthResolver.hasNativeCredentialProperties(properties) + && S3CompatSignals.hasExplicitS3Request(properties)) { + throw new IllegalArgumentException("Native GCP authentication cannot be used with S3 properties."); + } + } + + @Override + public Set fallbackProviders() { + return Set.of("S3", "MINIO"); + } + @Override public Optional credentialCodec() { return Optional.of(new GcsCredentialCodec()); @@ -143,7 +157,7 @@ public boolean supports(Map properties) { || S3CompatSignals.isFsSupport(properties, FS_GCS_SUPPORT)) { return true; } - return S3CompatSignals.guessAllowed(properties) && S3CompatSignals.guessIsGcs(properties); + return S3CompatSignals.guessAllowed(properties) && GcsAuthResolver.guessIsGcs(properties); } @Override @@ -172,7 +186,7 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { - return !S3CompatSignals.hasExplicitS3Request(properties) && S3CompatSignals.guessIsGcs(properties); + return !S3CompatSignals.hasExplicitS3Request(properties) && GcsAuthResolver.guessIsGcs(properties); } @Override diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java index 022618003d4f92..1d088e10bd786d 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java @@ -185,6 +185,20 @@ void sensitivePropertyKeys_coverSecretAliasesButNotAccessKey() { Assertions.assertFalse(keys.contains("gs.access_key"), keys.toString()); } + @Test + void validatesOwnedCredentialsEvenWhenS3IsSelected() { + for (Map selection : java.util.List.of(Map.of("provider", "S3"), + Map.of("fs.s3.support", "true"))) { + Map props = new HashMap<>(selection); + props.put("gs.credential_provider_type", "DEFAULT"); + Assertions.assertThrows(IllegalArgumentException.class, () -> provider.validateProperties(props)); + } + Assertions.assertDoesNotThrow(() -> provider.validateProperties(Map.of("provider", "S3"))); + Assertions.assertDoesNotThrow(() -> provider.validateProperties(Map.of("provider", "GCP"))); + Assertions.assertEquals(Set.of("S3", "MINIO"), provider.fallbackProviders()); + Assertions.assertTrue(provider.supportsGuess(Map.of("gs.impersonation_service_account", ""))); + } + @Test void name_isGcs() { Assertions.assertEquals("GCS", provider.name()); diff --git a/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java index 245f999117472b..f08e741dd39b2f 100644 --- a/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java @@ -89,10 +89,6 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { - // bindAll consults every provider, even after S3 or GCS has matched. - if (S3CompatSignals.hasExplicitS3Request(properties) || S3CompatSignals.guessIsGcs(properties)) { - return false; - } // Verbatim port of fe-core MinioProperties.guessIsMe: MinIO is the "any other // S3-compatible" fallback — it claims the map iff none of Azure/COS/OSS/S3 would claim // it AND at least one identifier key is present. The sibling guesses are replicated diff --git a/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java index eab1db44030cbb..1058dedc7fab0b 100644 --- a/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java @@ -195,16 +195,4 @@ void sensitivePropertyKeys_coverSecretAliasesButNotAccessKey() { void name_isMinio() { Assertions.assertEquals("MINIO", provider.name()); } - - @Test - void supportsGuess_doesNotClaimExplicitS3OrRegionalGcs() { - for (String endpoint : new String[] {"https://storage.googleapis.com:443/", - "https://storage.us-east1.rep.googleapis.com", "https://us-east1-storage.googleapis.com"}) { - Map props = new HashMap<>(); - props.put("s3.endpoint", endpoint); - Assertions.assertFalse(provider.supportsGuess(props)); - props.put("provider", "S3"); - Assertions.assertFalse(provider.supportsGuess(props)); - } - } } diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml b/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml index 78202d93f10138..ae5ef4de32e4db 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-s3-base/pom.xml @@ -37,11 +37,6 @@ under the License. Not a plugin itself: no provider, no META-INF/services. - - org.apache.doris - fe-filesystem-gcs-common - ${revision} - org.apache.doris fe-filesystem-spi diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java index 1f42ad43528b74..b78a71759d930c 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java @@ -173,14 +173,18 @@ public static boolean guessAllowed(Map properties) { /** * Port of {@code GCSProperties.guessIsMe}: a non-blank {@code gs.endpoint}, or any endpoint alias - * whose URI host identifies a GCS global, regional, or virtual-hosted endpoint. + * whose value ends with {@code storage.googleapis.com}. * *

Deviation: legacy compares {@code key.toLowerCase()} against a set holding the un-lowercased * {@code "AWS_ENDPOINT"}/{@code "ENDPOINT"}, so those two aliases can never match there. Here the * key comparison is genuinely case-insensitive, which is what legacy clearly intended. */ public static boolean guessIsGcs(Map properties) { - return org.apache.doris.filesystem.gcs.auth.GcsAuthResolver.guessIsGcs(properties); + if (StringUtils.isNotBlank(properties.get(GCS_ENDPOINT_KEY))) { + return true; + } + String endpoint = endpointForGuessing(properties); + return endpoint != null && endpoint.toLowerCase(Locale.ROOT).endsWith(GCS_ENDPOINT_SUFFIX); } /** diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java index e565954620ed1a..dc6ca844360cf0 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java @@ -208,10 +208,6 @@ public static S3FileSystemProperties of(Map properties) { @Override public void validate() { - if (org.apache.doris.filesystem.gcs.auth.GcsAuthResolver.hasNativeCredentialProperties(rawProperties)) { - org.apache.doris.filesystem.gcs.auth.GcsAuthResolver.resolve(rawProperties); - throw new IllegalArgumentException("Native GCP authentication cannot be used with S3 properties."); - } new ParamRules() .requireTogether(new String[] {accessKey, secretKey}, "s3.access_key and s3.secret_key must be set together") diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java index 0418296c09b84d..46e8387c4d285f 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java @@ -429,10 +429,4 @@ void forceParsingByStandardUri_parsesS3SchemeAsStandardUrl() { Assertions.assertEquals("s3.us-west-2.amazonaws.com", properties.getEndpoint()); Assertions.assertEquals("us-west-2", properties.getRegion()); } - - @Test - void rejectsNativeGcpCredentialsInsteadOfFallingBackToAws() { - Assertions.assertThrows(IllegalArgumentException.class, () -> S3FileSystemProperties.of(Map.of( - "provider", "S3", "s3.region", "us-east-1", "gs.credential_provider_type", "DEFAULT"))); - } } diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java index e4a8d3b9c39656..a4f95a85fd0a66 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java @@ -110,16 +110,11 @@ public FileSystem create(S3FileSystemProperties properties) throws IOException { @Override public boolean supportsExplicit(Map properties) { - return S3CompatSignals.hasExplicitS3Request(properties); + return Boolean.parseBoolean(properties.getOrDefault(FS_S3_SUPPORT, "false")); } @Override public boolean supportsGuess(Map properties) { - // GCS can use its default endpoint with only an S3-compatible region property. Yield - // before the region fallback so bindPrimary/bindAll select the GCS authentication path. - if (!S3CompatSignals.hasExplicitS3Request(properties) && S3CompatSignals.guessIsGcs(properties)) { - return false; - } // Port of fe-core S3Properties.guessIsMe on raw props. String endpoint = null; for (String name : GUESS_ENDPOINT_NAMES) { diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java index 360a41ee838c78..c81f119386bc97 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java @@ -181,38 +181,6 @@ void supports_stillAcceptsUnmarkedGenericFallback() { Assertions.assertTrue(provider.supports(props)); } - @Test - void supportsGuess_yieldsToGcsWithoutAnExplicitEndpoint() { - for (Map.Entry hint : Map.of( - "provider", "GCP", - "gs.credential_provider_type", "DEFAULT", - "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com", - "warehouse", "gs://bucket/warehouse").entrySet()) { - Map props = new HashMap<>(); - props.put("s3.region", "us-east1"); - props.put("_STORAGE_TYPE_", "S3"); - props.put(hint.getKey(), hint.getValue()); - - Assertions.assertFalse(provider.supportsGuess(props), "must yield for " + hint); - } - } - - @Test - void supportsGuess_preservesS3RegionFallbackAndExplicitSelection() { - Map props = new HashMap<>(); - props.put("s3.region", "us-east-1"); - Assertions.assertTrue(provider.supportsGuess(props)); - - props.put("provider", "S3"); - props.put("warehouse", "gs://bucket/warehouse"); - Assertions.assertTrue(provider.supportsGuess(props)); - - props.remove("provider"); - props.put("fs.s3.support", "true"); - Assertions.assertTrue(provider.supportsExplicit(props)); - Assertions.assertTrue(provider.supportsGuess(props)); - } - @Test void bind_returnsValidatedS3FileSystemProperties() { Map props = new HashMap<>(); @@ -252,12 +220,4 @@ void sensitivePropertyKeysCoverSecretsButNotAccessKey() { Assertions.assertFalse(keys.contains("s3.access_key"), keys.toString()); Assertions.assertFalse(keys.contains("AWS_ACCESS_KEY"), keys.toString()); } - - @Test - void supportsExplicit_preservesS3DefaultChainAtGcsEndpoint() { - Map props = Map.of("provider", "s3", - "s3.endpoint", "https://storage.googleapis.com", "s3.region", "us-east1"); - Assertions.assertTrue(provider.supportsExplicit(props)); - Assertions.assertEquals("DEFAULT", provider.bind(props).toFileSystemKv().get("AWS_CREDENTIALS_PROVIDER_TYPE")); - } } diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java index 205d7c58a4f31b..a143185d3f07aa 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java @@ -61,6 +61,23 @@ default Optional resolveAuthentication(Map properties) { + } + + /** + * Names of general-purpose providers whose heuristic matches this provider supersedes when it + * also matches. Explicit selections are never suppressed. Names are case-insensitive; the + * relationship must be acyclic. Empty by default to preserve existing multi-provider bindings. + */ + default Set fallbackProviders() { + return Collections.emptySet(); + } + /** Optional codec for this provider's native BE/Cloud credentials. */ default Optional credentialCodec() { return Optional.empty(); diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt index ada0edd5d04ac4..d06e058591c2f1 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt @@ -21,6 +21,7 @@ org.apache.doris.filesystem.spi.FileSystemProvider#createUntyped(org.apache.dori org.apache.doris.filesystem.spi.FileSystemProvider#credentialCodec():java.util.Optional org.apache.doris.filesystem.spi.FileSystemProvider#credentialToProperties(java.util.Map):java.util.Map org.apache.doris.filesystem.spi.FileSystemProvider#description():java.lang.String +org.apache.doris.filesystem.spi.FileSystemProvider#fallbackProviders():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#modifiableCredentialPropertyKeys():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#name():java.lang.String org.apache.doris.filesystem.spi.FileSystemProvider#normalizeProperties(java.util.Map,java.util.Map):java.util.Map @@ -29,6 +30,7 @@ org.apache.doris.filesystem.spi.FileSystemProvider#sensitivePropertyKeys():java. org.apache.doris.filesystem.spi.FileSystemProvider#supports(java.util.Map):boolean org.apache.doris.filesystem.spi.FileSystemProvider#supportsExplicit(java.util.Map):boolean org.apache.doris.filesystem.spi.FileSystemProvider#supportsGuess(java.util.Map):boolean +org.apache.doris.filesystem.spi.FileSystemProvider#validateProperties(java.util.Map):void org.apache.doris.filesystem.spi.ObjFileSystem#capability(java.lang.Class):java.util.Optional org.apache.doris.filesystem.spi.ObjFileSystem#close():void org.apache.doris.filesystem.spi.ObjFileSystem#completeMultipartUpload(java.lang.String,java.lang.String,java.util.Map):void diff --git a/fe/fe-filesystem/pom.xml b/fe/fe-filesystem/pom.xml index d8a6e8609463da..1d6dd09d27f612 100644 --- a/fe/fe-filesystem/pom.xml +++ b/fe/fe-filesystem/pom.xml @@ -53,7 +53,7 @@ under the License. bumping all five families. See plan-doc/designs/2026-07-29-plugin-api-version-check-design.md. --> - 3.0 + 4.0 From 1889e1791dea65de60a0757e60959f0e61b5d33f Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 12 Oct 2026 17:55:49 +0800 Subject: [PATCH 31/47] [refactor](fe) Separate typed storage identities from wire adapters ### What problem does this PR solve? Related PR: #68540 Problem Summary: Native storage authentication used plugin-supplied wire codecs and opaque byte arrays, coupling GCS plugins to host-generated protobuf and Thrift classes. Return immutable typed identity descriptions from the filesystem API and translate them directly in a dedicated host-side fe-storage-protocol module. Keep property parsing, validation, credential loading and property restoration in the owning plugin; keep fe-core provider-neutral. Preserve resolved identity versus partial ALTER field presence, including explicit clears. Reject unknown wire identity sources and missing or ambiguous property owners. Remove GCS protocol dependencies and intermediate serialization. Bump filesystem API to 5.0 and connector API to 15.0 for the shared signature changes. S3, S3-base and MinIO remain unchanged relative to the PR base. ### Release note Plugins must be rebuilt against the updated shared APIs. Unknown persisted GCP identity source values now fail explicitly instead of silently falling back to default credentials. ### Check List (For Author) - Test: Clean Maven module tests passed: 389 filesystem API/SPI, GCS, S3-base, S3 and MinIO tests, including plugin classloader isolation. Checkstyle and connector import/metadata funnel checks passed. - Test: 24 additional protocol/core routing test methods passed in an independent harness with both Thrift 0.16 and 0.24 runtimes. The harness compiles actual sources using extracted current protocol definitions; it is not a full FE build. - Test limitation: Standard protocol/connector Maven tests are blocked by the local Thrift compiler/runtime mismatch; the standard FE test entry is blocked by local JAVA_HOME validation. Full FE validation remains for CI. - Behavior changed: Yes. Typed plugin API replaces map and byte-array credential transport; unsupported wire sources fail explicitly. Existing authentication routing and ALTER clear/absence semantics are preserved. - Does this need documentation: Yes. Updated filesystem README with ownership, dependency boundaries, extension requirements and API compatibility. Self-review: Scope is the authentication boundary and its tests/docs. Shared descriptors and the codec registry are immutable; no new mutable global state or configuration is introduced. Both Cloud protobuf and BE Thrift paths are covered. Wire schemas and persisted field meanings are unchanged; absent and explicit-clear fields have dedicated coverage. Classloader isolation is tested, provider-specific SDKs remain in plugins, and errors identify unsupported/ambiguous ownership without logging credential values. --- .../spi/ConnectorPluginSurfaceTest.java | 5 +- fe/fe-connector/pom.xml | 2 +- fe/fe-core/pom.xml | 5 + .../storage/CloudObjectStoreAdapter.java | 6 +- .../ObjectStorageCredentialAdapter.java | 57 +++----- .../datasource/storage/StorageAdapter.java | 20 +-- .../apache/doris/fs/GcsGuessRoutingTest.java | 29 ++++ .../ObjectStorageCredentialAdapterTest.java | 117 +++++++++++++++++ .../fs/ObjectStorageCredentialCodecTest.java | 124 ------------------ fe/fe-filesystem/README.md | 42 ++++-- .../filesystem/auth/GcpCredentialSpec.java | 73 +++++++++++ .../auth/NativeObjectStorageCredential.java | 30 +++++ .../auth/ObjectStorageAuthentication.java | 59 ++++++--- .../auth/ObjectStorageAuthenticationTest.java | 83 ++++++++++++ .../fe-filesystem-gcs-common/pom.xml | 2 +- .../filesystem/gcs/auth/GcpCredential.java | 2 +- fe/fe-filesystem/fe-filesystem-gcs/pom.xml | 14 -- .../filesystem/gcs/GcsCredentialCodec.java | 87 ------------ .../filesystem/gcs/GcsFileSystemProvider.java | 55 ++++---- .../gcs/GcsCredentialCodecTest.java | 113 ---------------- .../gcs/GcsCredentialDescriptionTest.java | 122 +++++++++++++++++ .../gcs/GcsFileSystemProviderTest.java | 10 +- .../filesystem/spi/FileSystemProvider.java | 13 +- .../spi/ObjectStorageCredentialCodec.java | 42 ------ .../spi/FileSystemPluginSurfaceTest.java | 4 +- ...FileSystemProviderTypedPropertiesTest.java | 5 +- .../resources/filesystem-plugin-surface.txt | 33 +++-- fe/fe-filesystem/pom.xml | 2 +- fe/fe-storage-protocol/pom.xml | 55 ++++++++ .../storage/protocol/GcpCredentialCodec.java | 86 ++++++++++++ .../ObjectStorageCredentialCodec.java | 33 +++++ .../ObjectStorageCredentialTranslator.java | 76 +++++++++++ ...ObjectStorageCredentialTranslatorTest.java | 116 ++++++++++++++++ fe/pom.xml | 1 + 34 files changed, 1001 insertions(+), 522 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialAdapterTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialSpec.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/NativeObjectStorageCredential.java create mode 100644 fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/ObjectStorageAuthenticationTest.java delete mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java delete mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java create mode 100644 fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialDescriptionTest.java delete mode 100644 fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java create mode 100644 fe/fe-storage-protocol/pom.xml create mode 100644 fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/GcpCredentialCodec.java create mode 100644 fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialCodec.java create mode 100644 fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslator.java create mode 100644 fe/fe-storage-protocol/src/test/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslatorTest.java diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index b20c00705918b8..645f6241b66b99 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -87,9 +87,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException Assertions.assertNotNull(in, "missing connector plugin API version resource"); version.load(in); } - // Major 14 adds the filesystem authentication SPI changes on top of major 13's - // textual write-input semantics on ConnectorColumn. - Assertions.assertEquals("14.0", version.getProperty("api.version")); + // Major 15 replaces filesystem credential maps with typed, protocol-independent descriptions. + Assertions.assertEquals("15.0", version.getProperty("api.version")); } /** Root entry points plus provider/handle types returned to connector plugins. */ diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index 913f9ba889077e..c207b98365f253 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -55,7 +55,7 @@ under the License. of the latter two means bumping this property as well (and fe-extension-spi means bumping all five families). --> - 14.0 + 15.0 diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index 656b9847c76147..46ba126fea0457 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -193,6 +193,11 @@ under the License. + + org.apache.doris + fe-storage-protocol + ${revision} + ${project.groupId} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java index ee356faaf28052..a32356112bb088 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java @@ -130,10 +130,8 @@ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map properties) { builder.clearCredential(); StorageAdapter.resolveAuthentication(properties) - .filter(auth -> !auth.getCredentialUpdates().isEmpty()) - .ifPresent(auth -> ObjectStorageCredentialAdapter.fromCredential( - auth.getProvider(), auth.getCredentialUpdates()) - .applyTo(builder)); + .flatMap(ObjectStorageAuthentication::getCredentialUpdates) + .ifPresent(credential -> ObjectStorageCredentialAdapter.fromCredential(credential).applyTo(builder)); } private static boolean hasCredentialsProviderType(Map properties) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java index 097e2b84b2764d..de189be6d11c90 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/ObjectStorageCredentialAdapter.java @@ -19,70 +19,51 @@ import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; -import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; +import org.apache.doris.storage.protocol.ObjectStorageCredentialTranslator; import org.apache.doris.thrift.TCredential; import org.apache.doris.thrift.TObjStorageType; import org.apache.doris.thrift.TS3StorageParam; -import org.apache.thrift.TDeserializer; -import org.apache.thrift.TException; -import org.apache.thrift.protocol.TBinaryProtocol; - -import java.io.IOException; import java.util.Map; import java.util.Optional; -/** Transfers plugin-encoded credentials into core wire envelopes without interpreting provider fields. */ +/** Connects provider-resolved identities to the host protocol adapter without provider-specific logic. */ public final class ObjectStorageCredentialAdapter { - private final ObjectStorageCredentialCodec codec; - private final Map fields; + private final NativeObjectStorageCredential credential; - private ObjectStorageCredentialAdapter(ObjectStorageCredentialCodec codec, Map fields) { - this.codec = codec; - this.fields = Map.copyOf(fields); + private ObjectStorageCredentialAdapter(NativeObjectStorageCredential credential) { + this.credential = credential; } public static Optional fromAuthentication( Optional authentication) { - return authentication.filter(ObjectStorageAuthentication::isNative) - .map(auth -> fromCredential(auth.getProvider(), auth.getCredential())); + return authentication.flatMap(ObjectStorageAuthentication::getCredential) + .map(ObjectStorageCredentialAdapter::fromCredential); } - public static ObjectStorageCredentialAdapter fromCredential(String provider, Map fields) { - return new ObjectStorageCredentialAdapter(StorageAdapter.credentialCodec(provider), fields); + public static ObjectStorageCredentialAdapter fromCredential(NativeObjectStorageCredential credential) { + return new ObjectStorageCredentialAdapter(credential); } public static Map toProperties(String provider, ObjectStoreCredentialPB credential) { - try { - return StorageAdapter.credentialCodec(provider).toProperties(credential.toByteArray()); - } catch (IOException e) { - throw new IllegalArgumentException("Cannot decode native object-storage credentials", e); - } + return StorageAdapter.credentialToProperties(ObjectStorageCredentialTranslator.fromCloud(provider, credential)); } public void applyTo(ObjectStoreInfoPB.Builder builder) { - try { - ObjectStoreCredentialPB credential = ObjectStoreCredentialPB.parseFrom(codec.toCloudCredential(fields)); - if (!builder.hasProvider()) { - builder.setProvider(ObjectStoreInfoPB.Provider.valueOf(codec.provider())); - } - builder.setCredential(credential); - } catch (IOException e) { - throw new IllegalArgumentException("Cannot encode native object-storage credentials for Cloud", e); + ObjectStoreCredentialPB encoded = ObjectStorageCredentialTranslator.toCloud(credential); + if (!builder.hasProvider()) { + builder.setProvider(ObjectStoreInfoPB.Provider.valueOf(credential.getProvider())); } + builder.setCredential(encoded); } public void applyTo(TS3StorageParam param) { - try { - TCredential credential = new TCredential(); - new TDeserializer(new TBinaryProtocol.Factory()).deserialize(credential, codec.toThriftCredential(fields)); - if (!param.isSetProvider()) { - param.setProvider(TObjStorageType.valueOf(codec.provider())); - } - param.setCredential(credential); - } catch (IOException | TException e) { - throw new IllegalArgumentException("Cannot encode native object-storage credentials for BE", e); + TCredential encoded = ObjectStorageCredentialTranslator.toThrift(credential); + if (!param.isSetProvider()) { + param.setProvider(TObjStorageType.valueOf(credential.getProvider())); } + param.setCredential(encoded); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index b041a5c48ae59a..1ddde0b6883a8c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -19,13 +19,13 @@ import org.apache.doris.common.Config; import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.FileSystemProperties; import org.apache.doris.filesystem.properties.FsCacheKeys; import org.apache.doris.filesystem.properties.HadoopStorageProperties; import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.filesystem.spi.FileSystemProvider; -import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; import org.apache.doris.foundation.property.StoragePropertiesException; import org.apache.doris.foundation.security.ExecutionAuthenticator; import org.apache.doris.fs.FileSystemPluginManager; @@ -177,22 +177,22 @@ public static boolean isClearableProperty(String key) { return manager().getProviders().stream().anyMatch(provider -> provider.clearablePropertyKeys().contains(key)); } - /** Resolve a native wire codec through the registered filesystem providers. */ - public static ObjectStorageCredentialCodec credentialCodec(String providerName) { - ObjectStorageCredentialCodec selected = null; + /** Restore user properties through the plugin owning the decoded identity description. */ + public static Map credentialToProperties(NativeObjectStorageCredential credential) { + Map selected = null; for (FileSystemProvider provider : manager().getProviders()) { - Optional codec = provider.credentialCodec(); - if (codec.isPresent() && codec.get().provider().equalsIgnoreCase(providerName)) { + Optional> properties = provider.credentialToProperties(credential); + if (properties.isPresent()) { if (selected != null) { throw new StoragePropertiesException( - "Multiple credential codecs for provider '" + providerName + "'"); + "Multiple credential property owners for provider: " + credential.getProvider()); } - selected = codec.get(); + selected = properties.get(); } } if (selected == null) { - throw new StoragePropertiesException("Native credential codec for provider '" + providerName - + "' is not available"); + throw new StoragePropertiesException("Filesystem plugin for credential properties is not available: " + + credential.getProvider()); } return selected; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java index d62bf7b7acc699..e44e12a2e01953 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java @@ -17,7 +17,12 @@ package org.apache.doris.fs; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; +import org.apache.doris.datasource.storage.ObjectStorageCredentialAdapter; +import org.apache.doris.datasource.storage.S3ThriftAdapter; import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.thrift.TS3StorageParam; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -110,6 +115,30 @@ public void testNativeCredentialsCannotBeIgnoredByS3Selection() { "gs.credential_provider_type", "DEFAULT"))); } + @Test + public void testTypedIdentityConnectsPluginCloudAndBeWithoutLosingAlterPresence() { + Map properties = Map.of("provider", "GCP", + "gs.credential_provider_type", "COMPUTE_ENGINE", + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"); + ObjectStoreInfoPB.Builder cloud = CloudObjectStoreAdapter.getObjStoreInfoPB(properties); + TS3StorageParam thrift = S3ThriftAdapter.getS3TStorageParam(properties); + Assertions.assertEquals("COMPUTE_ENGINE", cloud.getCredential().getGcpCredential() + .getCredentialProviderType().name()); + Assertions.assertEquals("COMPUTE_ENGINE", thrift.getCredential().getGcpCredential() + .getCredentialProviderType().name()); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", thrift.getCredential() + .getGcpCredential().getImpersonationServiceAccount()); + Assertions.assertEquals(Map.of("gs.credential_provider_type", "COMPUTE_ENGINE", + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"), + ObjectStorageCredentialAdapter.toProperties("GCP", cloud.getCredential())); + CloudObjectStoreAdapter.applyCredentialPatch(cloud, Map.of("gs.impersonation_service_account", "")); + Assertions.assertFalse(cloud.getCredential().getGcpCredential().hasCredentialProviderType()); + Assertions.assertTrue(cloud.getCredential().getGcpCredential().hasImpersonationServiceAccount()); + Assertions.assertEquals("", cloud.getCredential().getGcpCredential().getImpersonationServiceAccount()); + CloudObjectStoreAdapter.applyCredentialPatch(cloud, Map.of("provider", "GCP")); + Assertions.assertFalse(cloud.hasCredential()); + } + private static Map properties(String endpoint) { Map props = new HashMap<>(); props.put("s3.endpoint", endpoint); diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialAdapterTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialAdapterTest.java new file mode 100644 index 00000000000000..8baaf550536d74 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialAdapterTest.java @@ -0,0 +1,117 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.fs; + +import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; +import org.apache.doris.datasource.storage.ObjectStorageCredentialAdapter; +import org.apache.doris.datasource.storage.StorageAdapter; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.GcpCredentialSpec; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; +import org.apache.doris.filesystem.properties.FileSystemProperties; +import org.apache.doris.filesystem.spi.FileSystemProvider; +import org.apache.doris.foundation.property.StoragePropertiesException; +import org.apache.doris.thrift.TObjStorageType; +import org.apache.doris.thrift.TS3StorageParam; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Map; +import java.util.Optional; + +class ObjectStorageCredentialAdapterTest { + @AfterEach + void restoreBuiltinRegistry() { + StorageAdapter.initPluginManager(null); + } + + @Test + void hostTranslatesTypedCredentialsAndPluginOwnsPropertyNames() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(provider("GCS")); + StorageAdapter.initPluginManager(manager); + GcpCredentialSpec spec = new GcpCredentialSpec(GcpCredentialSpec.ProviderType.COMPUTE_ENGINE, null); + ObjectStorageCredentialAdapter adapter = ObjectStorageCredentialAdapter.fromCredential(spec); + ObjectStoreInfoPB.Builder cloud = ObjectStoreInfoPB.newBuilder(); + adapter.applyTo(cloud); + Assertions.assertEquals(ObjectStoreInfoPB.Provider.GCP, cloud.getProvider()); + Assertions.assertEquals("COMPUTE_ENGINE", cloud.getCredential().getGcpCredential() + .getCredentialProviderType().name()); + TS3StorageParam thrift = new TS3StorageParam(); + adapter.applyTo(thrift); + Assertions.assertEquals(TObjStorageType.GCP, thrift.getProvider()); + Assertions.assertEquals("COMPUTE_ENGINE", thrift.getCredential().getGcpCredential() + .getCredentialProviderType().name()); + Assertions.assertEquals(Map.of("plugin.identity", "COMPUTE_ENGINE"), + ObjectStorageCredentialAdapter.toProperties("GCP", cloud.getCredential())); + } + + @Test + void alterDoesNotInventSourceOrDropExplicitClear() { + ObjectStoreInfoPB.Builder cloud = ObjectStoreInfoPB.newBuilder(); + TS3StorageParam thrift = new TS3StorageParam(); + ObjectStorageCredentialAdapter adapter = ObjectStorageCredentialAdapter.fromCredential( + new GcpCredentialSpec(null, "")); + adapter.applyTo(cloud); + adapter.applyTo(thrift); + Assertions.assertFalse(cloud.getCredential().getGcpCredential().hasCredentialProviderType()); + Assertions.assertFalse(thrift.getCredential().getGcpCredential().isSetCredentialProviderType()); + Assertions.assertTrue(cloud.getCredential().getGcpCredential().hasImpersonationServiceAccount()); + Assertions.assertTrue(thrift.getCredential().getGcpCredential().isSetImpersonationServiceAccount()); + Assertions.assertEquals("", cloud.getCredential().getGcpCredential().getImpersonationServiceAccount()); + Assertions.assertEquals("", thrift.getCredential().getGcpCredential().getImpersonationServiceAccount()); + } + + @Test + void missingOrAmbiguousPropertyOwnerFailsExplicitly() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + StorageAdapter.initPluginManager(manager); + GcpCredentialSpec spec = new GcpCredentialSpec(GcpCredentialSpec.ProviderType.DEFAULT, null); + Assertions.assertThrows(StoragePropertiesException.class, () -> StorageAdapter.credentialToProperties(spec)); + manager.registerProvider(provider("first")); + manager.registerProvider(provider("second")); + Assertions.assertThrows(StoragePropertiesException.class, () -> StorageAdapter.credentialToProperties(spec)); + } + + private FileSystemProvider provider(String name) { + return new FileSystemProvider() { + @Override + public String name() { + return name; + } + + @Override + public boolean supports(Map properties) { + return false; + } + + @Override + public FileSystem create(Map properties) { + throw new UnsupportedOperationException("No filesystem I/O in credential conversion"); + } + + @Override + public Optional> credentialToProperties(NativeObjectStorageCredential credential) { + GcpCredentialSpec spec = (GcpCredentialSpec) credential; + return Optional.of(Map.of("plugin.identity", spec.getProviderType().orElseThrow().name())); + } + }; + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java deleted file mode 100644 index ae4e50972754a0..00000000000000 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/ObjectStorageCredentialCodecTest.java +++ /dev/null @@ -1,124 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.fs; - -import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; -import org.apache.doris.cloud.proto.Cloud.ObjectStoreInfoPB; -import org.apache.doris.datasource.storage.ObjectStorageCredentialAdapter; -import org.apache.doris.datasource.storage.StorageAdapter; -import org.apache.doris.filesystem.FileSystem; -import org.apache.doris.filesystem.properties.FileSystemProperties; -import org.apache.doris.filesystem.spi.FileSystemProvider; -import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; -import org.apache.doris.foundation.property.StoragePropertiesException; - -import com.google.protobuf.UnknownFieldSet; -import org.junit.jupiter.api.AfterEach; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -import java.util.Map; -import java.util.Optional; - -class ObjectStorageCredentialCodecTest { - @AfterEach - void restoreBuiltinRegistry() { - StorageAdapter.initPluginManager(null); - } - - @Test - void registeredPluginOwnsTheWireFieldsAndPropertyMapping() { - FileSystemPluginManager manager = new FileSystemPluginManager(); - RecordingCodec codec = new RecordingCodec(); - manager.registerProvider(provider("external-codec-plugin", codec)); - StorageAdapter.initPluginManager(manager); - Assertions.assertSame(codec, StorageAdapter.credentialCodec("oss")); - Map fields = Map.of("plugin-owned-field", "value"); - ObjectStoreInfoPB.Builder builder = ObjectStoreInfoPB.newBuilder(); - ObjectStorageCredentialAdapter.fromCredential("OSS", fields).applyTo(builder); - Assertions.assertEquals(ObjectStoreInfoPB.Provider.OSS, builder.getProvider()); - Assertions.assertEquals(fields, codec.fields); - Assertions.assertEquals(codec.envelope, builder.getCredential()); - Assertions.assertEquals(Map.of("plugin.property", "restored"), - ObjectStorageCredentialAdapter.toProperties("OSS", builder.getCredential())); - } - - @Test - void missingOrAmbiguousCodecFailsWithoutFallingBackToAnotherProvider() { - FileSystemPluginManager manager = new FileSystemPluginManager(); - StorageAdapter.initPluginManager(manager); - Assertions.assertThrows(StoragePropertiesException.class, () -> StorageAdapter.credentialCodec("OSS")); - manager.registerProvider(provider("first", new RecordingCodec())); - manager.registerProvider(provider("second", new RecordingCodec())); - Assertions.assertThrows(StoragePropertiesException.class, () -> StorageAdapter.credentialCodec("OSS")); - } - - private FileSystemProvider provider(String name, ObjectStorageCredentialCodec codec) { - return new FileSystemProvider() { - @Override - public String name() { - return name; - } - - @Override - public boolean supports(Map properties) { - return false; - } - - @Override - public FileSystem create(Map properties) { - throw new UnsupportedOperationException("No filesystem I/O in credential conversion"); - } - - @Override - public Optional credentialCodec() { - return Optional.of(codec); - } - }; - } - - private static class RecordingCodec implements ObjectStorageCredentialCodec { - // The kernel must transfer even fields it does not interpret. - private final ObjectStoreCredentialPB envelope = ObjectStoreCredentialPB.newBuilder() - .setUnknownFields(UnknownFieldSet.newBuilder().addField(100, - UnknownFieldSet.Field.newBuilder().addVarint(42).build()).build()).build(); - private Map fields; - - @Override - public String provider() { - return "OSS"; - } - - @Override - public byte[] toCloudCredential(Map credential) { - fields = credential; - return envelope.toByteArray(); - } - - @Override - public byte[] toThriftCredential(Map credential) { - throw new UnsupportedOperationException("This fixture only supplies Cloud credentials"); - } - - @Override - public Map toProperties(byte[] cloudCredential) { - Assertions.assertArrayEquals(envelope.toByteArray(), cloudCredential); - return Map.of("plugin.property", "restored"); - } - } -} diff --git a/fe/fe-filesystem/README.md b/fe/fe-filesystem/README.md index 127cf6e8cc029f..e189f3c6562205 100644 --- a/fe/fe-filesystem/README.md +++ b/fe/fe-filesystem/README.md @@ -141,19 +141,37 @@ At runtime, `DirectoryPluginRuntimeManager` creates an isolated `ClassLoader` pe `software.amazon.awssdk.*`, `org.apache.hadoop.*`) are loaded parent-first to avoid cross- ClassLoader cast failures. -### Native object-storage credential codecs +### Native object-storage identities and protocol adapters -Providers with native credentials expose an optional `ObjectStorageCredentialCodec` through -`FileSystemProvider.credentialCodec()`. The plugin owns provider-specific field names and converts -resolved credentials to the existing Cloud protobuf and BE Thrift envelopes. Only JDK maps and byte -arrays cross this SPI; the API/SPI modules retain their dependency-free protocol boundary. -`fe-core` selects the registered codec and transfers the envelopes without inspecting provider fields. -The same codec restores properties for SHOW CREATE and Cloud stages. Partial ALTER maps must retain -the distinction between an absent field and an explicitly empty value. +The credential path has three layers with one-way dependencies: -The GCS codec uses host-provided `fe-grpc` and `fe-thrift` generated types (provided dependencies), -without depending on `fe-core`. Filesystem plugin API 4.0 includes this extension point; plugins must -be rebuilt for the matching host API. +``` +GCS plugin -> filesystem API identity description <- fe-storage-protocol -> fe-grpc / fe-thrift + ^ ^ + | | + fe-core ---------------------+ +``` + +The GCS plugin owns user property aliases, validation and authentication. It returns an immutable +`GcpCredentialSpec` through `ObjectStorageAuthentication`. The shared API contains only data types; +it imports no SDK or generated protocol classes. Resolved credentials and partial ALTER updates are +separate values. An absent field stays absent, while an explicitly empty impersonation account clears +that setting. HMAC and anonymous modes do not contain native credentials. + +The host-side `fe-storage-protocol` module converts those descriptions directly to Cloud protobuf and +BE Thrift objects. It owns wire-schema mapping and depends on the filesystem API and generated +protocol modules, never on filesystem plugin implementations. There is no intermediate serialization, +reflection-based field mapping, or plugin-supplied wire codec. `fe-core` invokes the generic translator +and attaches the result to its request envelopes without interpreting GCP fields. + +For SHOW CREATE and Cloud stages, the protocol adapter decodes a description and the owning plugin's +`credentialToProperties()` restores user property names. Unknown wire providers, mismatched description +types, and missing or ambiguous property owners fail explicitly. + +Adding a native wire variant requires a data description, a mapping in `fe-storage-protocol`, and +matching BE/Cloud support. Filesystem plugins cannot independently extend the receiver's wire schema. +The filesystem plugin API is 5.0 and connector plugin API is 15.0 because shared authentication types +changed. Plugins must be rebuilt against the matching host APIs. ### Overlapping provider matches @@ -169,7 +187,7 @@ perform no I/O. Forced provider binding validates with the effective provider na GCS owns its endpoint/credential recognition and declares S3 and MinIO as heuristic fallbacks. S3 and its shared protocol library do not depend on GCS authentication classes. Filesystem plugin -API 4.0 adds these two default hooks; all plugins must be rebuilt for that host API. +API 5.0 includes these two default hooks; all plugins must be rebuilt for that host API. ### S3 Delegation Pattern diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialSpec.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialSpec.java new file mode 100644 index 00000000000000..5711ccc184a79e --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/GcpCredentialSpec.java @@ -0,0 +1,73 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import java.util.Objects; +import java.util.Optional; + +/** + * GCP identity description shared by the GCS plugin and the host protocol adapter. + * Validation of user properties and credential loading belong to the GCS plugin. + * Null fields mean absent; an empty impersonation account is an explicit ALTER clear. + */ +public final class GcpCredentialSpec implements NativeObjectStorageCredential { + public enum ProviderType { + DEFAULT, COMPUTE_ENGINE + } + + private final ProviderType providerType; + private final String impersonationServiceAccount; + + public GcpCredentialSpec(ProviderType providerType, String impersonationServiceAccount) { + this.providerType = providerType; + this.impersonationServiceAccount = impersonationServiceAccount; + } + + @Override + public String getProvider() { + return "GCP"; + } + + public Optional getProviderType() { + return Optional.ofNullable(providerType); + } + + public Optional getImpersonationServiceAccount() { + return Optional.ofNullable(impersonationServiceAccount); + } + + @Override + public boolean isEmpty() { + return providerType == null && impersonationServiceAccount == null; + } + + @Override + public boolean equals(Object other) { + if (!(other instanceof GcpCredentialSpec)) { + return false; + } + GcpCredentialSpec that = (GcpCredentialSpec) other; + return providerType == that.providerType + && Objects.equals(impersonationServiceAccount, that.impersonationServiceAccount); + } + + @Override + public int hashCode() { + return Objects.hash(providerType, impersonationServiceAccount); + } +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/NativeObjectStorageCredential.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/NativeObjectStorageCredential.java new file mode 100644 index 00000000000000..4a83e8e9078ed2 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/NativeObjectStorageCredential.java @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +/** + * Immutable, protocol-independent description of a native object-storage identity. + * Implementations contain data only: no clients, credential loading, or wire serialization. + * New wire variants also require support in the host protocol adapter and the receiving BE/Cloud. + */ +public interface NativeObjectStorageCredential { + String getProvider(); + + /** True for an ALTER patch with no supplied fields. */ + boolean isEmpty(); +} diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java index 27e9251acf1980..5448da998cd6c9 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/auth/ObjectStorageAuthentication.java @@ -17,27 +17,44 @@ package org.apache.doris.filesystem.auth; -import java.util.Map; - -/** - * Provider-resolved authentication, with no property parsing or SDK dependencies. - * - *

The provider identifies the wire credential variant (for example, GCP). Credential map keys - * are fields of that variant, not user-facing configuration keys. Only protocol adapters interpret - * these fields; callers use the authentication mode and pass the maps through unchanged.

- */ +import java.util.Objects; +import java.util.Optional; + +/** Provider-resolved authentication, independent of user property aliases, SDKs and wire formats. */ public final class ObjectStorageAuthentication { private final String provider; private final boolean anonymous; - private final Map credential; - private final Map credentialUpdates; + private final NativeObjectStorageCredential credential; + private final NativeObjectStorageCredential credentialUpdates; - public ObjectStorageAuthentication(String provider, boolean anonymous, Map credential, - Map credentialUpdates) { - this.provider = provider; + private ObjectStorageAuthentication(String provider, boolean anonymous, + NativeObjectStorageCredential credential, NativeObjectStorageCredential credentialUpdates) { + this.provider = Objects.requireNonNull(provider, "provider"); this.anonymous = anonymous; - this.credential = Map.copyOf(credential); - this.credentialUpdates = Map.copyOf(credentialUpdates); + this.credential = credential; + this.credentialUpdates = credentialUpdates; + } + + public static ObjectStorageAuthentication nativeAuthentication(NativeObjectStorageCredential credential, + NativeObjectStorageCredential updates) { + Objects.requireNonNull(credential, "credential"); + Objects.requireNonNull(updates, "updates"); + if (credential.isEmpty()) { + throw new IllegalArgumentException("Resolved native credentials must not be empty"); + } + if (!credential.getProvider().equals(updates.getProvider())) { + throw new IllegalArgumentException("Credential and ALTER patch must identify the same provider"); + } + return new ObjectStorageAuthentication(credential.getProvider(), false, credential, + updates.isEmpty() ? null : updates); + } + + public static ObjectStorageAuthentication hmac(String provider) { + return new ObjectStorageAuthentication(provider, false, null, null); + } + + public static ObjectStorageAuthentication anonymous(String provider) { + return new ObjectStorageAuthentication(provider, true, null, null); } public String getProvider() { @@ -49,15 +66,15 @@ public boolean isAnonymous() { } public boolean isNative() { - return !credential.isEmpty(); + return credential != null; } - public Map getCredential() { - return credential; + public Optional getCredential() { + return Optional.ofNullable(credential); } /** Only explicitly supplied native fields, including empty values that clear persisted fields. */ - public Map getCredentialUpdates() { - return credentialUpdates; + public Optional getCredentialUpdates() { + return Optional.ofNullable(credentialUpdates); } } diff --git a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/ObjectStorageAuthenticationTest.java b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/ObjectStorageAuthenticationTest.java new file mode 100644 index 00000000000000..8458e06f43af7b --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/auth/ObjectStorageAuthenticationTest.java @@ -0,0 +1,83 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.auth; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +class ObjectStorageAuthenticationTest { + @Test + void absentAndClearedFieldsAreDifferentValues() { + GcpCredentialSpec absent = new GcpCredentialSpec(null, null); + GcpCredentialSpec cleared = new GcpCredentialSpec(null, ""); + Assertions.assertTrue(absent.isEmpty()); + Assertions.assertFalse(cleared.isEmpty()); + Assertions.assertNotEquals(absent, cleared); + Assertions.assertTrue(absent.getImpersonationServiceAccount().isEmpty()); + Assertions.assertEquals("", cleared.getImpersonationServiceAccount().orElseThrow()); + Assertions.assertEquals(cleared, new GcpCredentialSpec(null, "")); + Assertions.assertEquals(cleared.hashCode(), new GcpCredentialSpec(null, "").hashCode()); + } + + @Test + void resolvedIdentityAndPartialUpdateHaveSeparatePresence() { + GcpCredentialSpec resolved = new GcpCredentialSpec(GcpCredentialSpec.ProviderType.DEFAULT, null); + ObjectStorageAuthentication initial = ObjectStorageAuthentication.nativeAuthentication( + resolved, new GcpCredentialSpec(null, null)); + Assertions.assertTrue(initial.isNative()); + Assertions.assertFalse(initial.isAnonymous()); + Assertions.assertEquals("GCP", initial.getProvider()); + Assertions.assertEquals(resolved, initial.getCredential().orElseThrow()); + Assertions.assertTrue(initial.getCredentialUpdates().isEmpty()); + ObjectStorageAuthentication alter = ObjectStorageAuthentication.nativeAuthentication( + resolved, new GcpCredentialSpec(null, "")); + Assertions.assertEquals(new GcpCredentialSpec(null, ""), alter.getCredentialUpdates().orElseThrow()); + } + + @Test + void invalidNativeAuthenticationCannotBeConstructed() { + Assertions.assertThrows(IllegalArgumentException.class, () -> ObjectStorageAuthentication.nativeAuthentication( + new GcpCredentialSpec(null, null), new GcpCredentialSpec(null, null))); + NativeObjectStorageCredential foreign = new NativeObjectStorageCredential() { + @Override + public String getProvider() { + return "OTHER"; + } + + @Override + public boolean isEmpty() { + return false; + } + }; + Assertions.assertThrows(IllegalArgumentException.class, () -> ObjectStorageAuthentication.nativeAuthentication( + new GcpCredentialSpec(GcpCredentialSpec.ProviderType.DEFAULT, null), foreign)); + } + + @Test + void nonNativeModesCannotContainNativeFields() { + ObjectStorageAuthentication hmac = ObjectStorageAuthentication.hmac("GCP"); + ObjectStorageAuthentication anonymous = ObjectStorageAuthentication.anonymous("GCP"); + Assertions.assertFalse(hmac.isAnonymous()); + Assertions.assertTrue(anonymous.isAnonymous()); + for (ObjectStorageAuthentication auth : new ObjectStorageAuthentication[] {hmac, anonymous}) { + Assertions.assertFalse(auth.isNative()); + Assertions.assertTrue(auth.getCredential().isEmpty()); + Assertions.assertTrue(auth.getCredentialUpdates().isEmpty()); + } + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml b/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml index 7dfeaa7e60ab22..b6357bbf3185b3 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/pom.xml @@ -31,7 +31,7 @@ under the License. fe-filesystem-gcs-common Doris GCS authentication and property rules - Provider implementation shared by GCS and S3-compatible plugins; not part of the host API. + GCS property and authentication rules shared by GCS and Iceberg; not part of the host API. org.junit.jupiter diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java index 14130bbc0ecf61..c32cd4d271a4f5 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcpCredential.java @@ -19,7 +19,7 @@ -/** GCP native credential shared by the Cloud protobuf and BE Thrift paths. */ +/** Validated GCP identity used by the GCS authentication implementation. */ public final class GcpCredential { public static final String CREDENTIAL_PROVIDER_TYPE = "gs.credential_provider_type"; public static final String IMPERSONATION_SERVICE_ACCOUNT = "gs.impersonation_service_account"; diff --git a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml index 4c66a21730509c..a2b6bdb6d69794 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/pom.xml +++ b/fe/fe-filesystem/fe-filesystem-gcs/pom.xml @@ -41,20 +41,6 @@ under the License. fe-filesystem-gcs-common ${revision} - - - org.apache.doris - fe-grpc - ${revision} - provided - - - org.apache.doris - fe-thrift - ${revision} - provided - com.google.auth google-auth-library-oauth2-http diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java deleted file mode 100644 index a988e9c1b1bda7..00000000000000 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsCredentialCodec.java +++ /dev/null @@ -1,87 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.filesystem.gcs; - -import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; -import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; -import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; -import org.apache.doris.thrift.TCredential; -import org.apache.doris.thrift.TGcpCredential; -import org.apache.doris.thrift.TGcpCredentialProviderType; - -import org.apache.thrift.TException; -import org.apache.thrift.TSerializer; -import org.apache.thrift.protocol.TBinaryProtocol; - -import java.io.IOException; -import java.util.HashMap; -import java.util.Map; - -/** Native GCP wire encoding belongs to the GCS plugin, alongside its authentication rules. */ -public final class GcsCredentialCodec implements ObjectStorageCredentialCodec { - @Override - public String provider() { - return "GCP"; - } - - @Override - public byte[] toCloudCredential(Map fields) { - GcpCredentialPB.Builder credential = GcpCredentialPB.newBuilder(); - if (fields.containsKey("credential_provider_type")) { - credential.setCredentialProviderType(GcpCredentialPB.CredentialProviderType.valueOf( - fields.get("credential_provider_type"))); - } - if (fields.containsKey("impersonation_service_account")) { - credential.setImpersonationServiceAccount(fields.get("impersonation_service_account")); - } - return ObjectStoreCredentialPB.newBuilder().setGcpCredential(credential).build().toByteArray(); - } - - @Override - public byte[] toThriftCredential(Map fields) throws IOException { - TGcpCredential credential = new TGcpCredential(); - if (fields.containsKey("credential_provider_type")) { - credential.setCredentialProviderType(TGcpCredentialProviderType.valueOf( - fields.get("credential_provider_type"))); - } - if (fields.containsKey("impersonation_service_account")) { - credential.setImpersonationServiceAccount(fields.get("impersonation_service_account")); - } - try { - return new TSerializer(new TBinaryProtocol.Factory()) - .serialize(new TCredential().setGcpCredential(credential)); - } catch (TException e) { - throw new IOException("Cannot encode GCS credentials for BE", e); - } - } - - @Override - public Map toProperties(byte[] cloudCredential) throws IOException { - ObjectStoreCredentialPB envelope = ObjectStoreCredentialPB.parseFrom(cloudCredential); - if (!envelope.hasGcpCredential()) { - throw new IOException("Missing GCP credentials in the Cloud credential envelope"); - } - GcpCredentialPB credential = envelope.getGcpCredential(); - Map fields = new HashMap<>(); - fields.put("credential_provider_type", credential.getCredentialProviderType().name()); - if (credential.hasImpersonationServiceAccount()) { - fields.put("impersonation_service_account", credential.getImpersonationServiceAccount()); - } - return new GcsFileSystemProvider().credentialToProperties(fields); - } -} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index b504613312fc0a..0b5920e888e7d7 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -18,6 +18,8 @@ package org.apache.doris.filesystem.gcs; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.GcpCredentialSpec; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.gcs.auth.GcpCredential; import org.apache.doris.filesystem.gcs.auth.GcsAuthResolver; @@ -25,7 +27,6 @@ import org.apache.doris.filesystem.s3.S3FileSystem; import org.apache.doris.filesystem.s3.S3FileSystemProperties; import org.apache.doris.filesystem.spi.FileSystemProvider; -import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; import org.apache.doris.foundation.property.ConnectorPropertiesUtils; import java.io.IOException; @@ -90,21 +91,21 @@ public Map normalizeProperties(Map properties, M @Override public Optional resolveAuthentication(Map properties) { return GcsAuthResolver.resolve(properties).map(auth -> { - Map credential = new HashMap<>(); - Map updates = new HashMap<>(); - auth.getNativeCredential().ifPresent(nativeCredential -> { - credential.put("credential_provider_type", nativeCredential.getCredentialProviderType().name()); - if (!nativeCredential.getImpersonationServiceAccount().isEmpty()) { - credential.put("impersonation_service_account", nativeCredential.getImpersonationServiceAccount()); - } - if (properties.containsKey(GcpCredential.CREDENTIAL_PROVIDER_TYPE)) { - updates.put("credential_provider_type", nativeCredential.getCredentialProviderType().name()); - } - if (properties.containsKey(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT)) { - updates.put("impersonation_service_account", nativeCredential.getImpersonationServiceAccount()); - } - }); - return new ObjectStorageAuthentication("GCP", auth.isAnonymous(), credential, updates); + if (auth.isAnonymous()) { + return ObjectStorageAuthentication.anonymous("GCP"); + } + if (auth.getNativeCredential().isEmpty()) { + return ObjectStorageAuthentication.hmac("GCP"); + } + GcpCredential nativeCredential = auth.getNativeCredential().orElseThrow(); + GcpCredentialSpec.ProviderType type = GcpCredentialSpec.ProviderType.valueOf( + nativeCredential.getCredentialProviderType().name()); + String principal = nativeCredential.getImpersonationServiceAccount(); + GcpCredentialSpec resolved = new GcpCredentialSpec(type, principal.isEmpty() ? null : principal); + GcpCredentialSpec updates = new GcpCredentialSpec( + properties.containsKey(GcpCredential.CREDENTIAL_PROVIDER_TYPE) ? type : null, + properties.containsKey(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT) ? principal : null); + return ObjectStorageAuthentication.nativeAuthentication(resolved, updates); }); } @@ -123,19 +124,17 @@ public Set fallbackProviders() { } @Override - public Optional credentialCodec() { - return Optional.of(new GcsCredentialCodec()); - } - - @Override - public Map credentialToProperties(Map credential) { - Map properties = new HashMap<>(); - properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.get("credential_provider_type")); - String principal = credential.get("impersonation_service_account"); - if (principal != null && !principal.isEmpty()) { - properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, principal); + public Optional> credentialToProperties(NativeObjectStorageCredential credential) { + if (!(credential instanceof GcpCredentialSpec)) { + return Optional.empty(); } - return properties; + GcpCredentialSpec gcp = (GcpCredentialSpec) credential; + Map properties = new HashMap<>(); + properties.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, + gcp.getProviderType().orElse(GcpCredentialSpec.ProviderType.DEFAULT).name()); + gcp.getImpersonationServiceAccount().filter(principal -> !principal.isEmpty()) + .ifPresent(principal -> properties.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, principal)); + return Optional.of(properties); } @Override diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java deleted file mode 100644 index d2a6efd89ab72b..00000000000000 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialCodecTest.java +++ /dev/null @@ -1,113 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.filesystem.gcs; - -import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; -import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; -import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; -import org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec; -import org.apache.doris.thrift.TCredential; -import org.apache.doris.thrift.TGcpCredential; -import org.apache.doris.thrift.TGcpCredentialProviderType; - -import org.apache.thrift.TDeserializer; -import org.apache.thrift.protocol.TBinaryProtocol; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -import java.io.IOException; -import java.util.Map; - -class GcsCredentialCodecTest { - private final GcsFileSystemProvider provider = new GcsFileSystemProvider(); - private final ObjectStorageCredentialCodec codec = provider.credentialCodec().orElseThrow(); - - @Test - void nativeCredentialsRoundTripThroughBothWireFormats() throws Exception { - for (String source : new String[] {"DEFAULT", "COMPUTE_ENGINE"}) { - Map properties = Map.of( - "provider", "GCP", - "gs.credential_provider_type", source, - "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"); - ObjectStorageAuthentication auth = provider.resolveAuthentication(properties).orElseThrow(); - Assertions.assertEquals(auth.getProvider(), codec.provider()); - byte[] cloudBytes = codec.toCloudCredential(auth.getCredential()); - GcpCredentialPB cloud = ObjectStoreCredentialPB.parseFrom(cloudBytes).getGcpCredential(); - TGcpCredential thrift = decodeThrift(auth.getCredential()); - Assertions.assertEquals(GcpCredentialPB.CredentialProviderType.valueOf(source), - cloud.getCredentialProviderType()); - Assertions.assertEquals(TGcpCredentialProviderType.valueOf(source), thrift.getCredentialProviderType()); - Assertions.assertEquals("target@test.iam.gserviceaccount.com", cloud.getImpersonationServiceAccount()); - Assertions.assertEquals(cloud.getImpersonationServiceAccount(), thrift.getImpersonationServiceAccount()); - Assertions.assertEquals(Map.of("gs.credential_provider_type", source, - "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"), - codec.toProperties(cloudBytes)); - } - } - - @Test - void partialAlterPreservesExplicitClearWithoutAddingProviderType() throws Exception { - ObjectStorageAuthentication auth = provider.resolveAuthentication(Map.of( - "provider", "GCP", "gs.impersonation_service_account", "")).orElseThrow(); - GcpCredentialPB cloud = ObjectStoreCredentialPB.parseFrom( - codec.toCloudCredential(auth.getCredentialUpdates())).getGcpCredential(); - TGcpCredential thrift = decodeThrift(auth.getCredentialUpdates()); - Assertions.assertFalse(cloud.hasCredentialProviderType()); - Assertions.assertFalse(thrift.isSetCredentialProviderType()); - Assertions.assertTrue(cloud.hasImpersonationServiceAccount()); - Assertions.assertTrue(thrift.isSetImpersonationServiceAccount()); - Assertions.assertEquals("", cloud.getImpersonationServiceAccount()); - Assertions.assertEquals("", thrift.getImpersonationServiceAccount()); - } - - @Test - void omittedImpersonationStaysAbsent() throws Exception { - Map fields = Map.of("credential_provider_type", "DEFAULT"); - byte[] cloudBytes = codec.toCloudCredential(fields); - Assertions.assertFalse(ObjectStoreCredentialPB.parseFrom(cloudBytes) - .getGcpCredential().hasImpersonationServiceAccount()); - Assertions.assertFalse(decodeThrift(fields).isSetImpersonationServiceAccount()); - Assertions.assertEquals(Map.of("gs.credential_provider_type", "DEFAULT"), codec.toProperties(cloudBytes)); - } - - @Test - void invalidEnvelopeFailsInsteadOfSelectingDefaultCredentials() { - Assertions.assertThrows(IOException.class, - () -> codec.toProperties(ObjectStoreCredentialPB.getDefaultInstance().toByteArray())); - Assertions.assertThrows(IOException.class, () -> codec.toProperties(new byte[] {(byte) 0xff})); - Assertions.assertThrows(IllegalArgumentException.class, - () -> codec.toCloudCredential(Map.of("credential_provider_type", "UNKNOWN"))); - Assertions.assertThrows(IllegalArgumentException.class, - () -> codec.toThriftCredential(Map.of("credential_provider_type", "UNKNOWN"))); - } - - @Test - void hmacAndAnonymousAuthenticationDoNotProduceNativeCredentials() { - Assertions.assertFalse(provider.resolveAuthentication(Map.of("provider", "GCP", - "s3.access_key", "ak", "s3.secret_key", "sk")).orElseThrow().isNative()); - Assertions.assertFalse(provider.resolveAuthentication(Map.of("provider", "GCP", - "gs.credential_provider_type", "ANONYMOUS")).orElseThrow().isNative()); - } - - private TGcpCredential decodeThrift(Map fields) throws Exception { - TCredential envelope = new TCredential(); - new TDeserializer(new TBinaryProtocol.Factory()).deserialize(envelope, codec.toThriftCredential(fields)); - Assertions.assertTrue(envelope.isSetGcpCredential()); - return envelope.getGcpCredential(); - } -} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialDescriptionTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialDescriptionTest.java new file mode 100644 index 00000000000000..8c82b703ef6418 --- /dev/null +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsCredentialDescriptionTest.java @@ -0,0 +1,122 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.filesystem.gcs; + +import org.apache.doris.filesystem.auth.GcpCredentialSpec; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; +import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; +import org.apache.doris.filesystem.gcs.auth.GcsAuthResolver; +import org.apache.doris.filesystem.spi.FileSystemProvider; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.net.URL; +import java.net.URLClassLoader; +import java.util.Map; + +class GcsCredentialDescriptionTest { + private final GcsFileSystemProvider provider = new GcsFileSystemProvider(); + + @Test + void nativeIdentityRoundTripsThroughUserPropertiesWithoutWireTypes() { + for (GcpCredentialSpec.ProviderType type : GcpCredentialSpec.ProviderType.values()) { + Map properties = Map.of("gs.credential_provider_type", type.name(), + "gs.impersonation_service_account", "target@test.iam.gserviceaccount.com"); + ObjectStorageAuthentication auth = provider.resolveAuthentication(properties).orElseThrow(); + GcpCredentialSpec expected = new GcpCredentialSpec(type, "target@test.iam.gserviceaccount.com"); + Assertions.assertEquals(expected, auth.getCredential().orElseThrow()); + Assertions.assertEquals(expected, auth.getCredentialUpdates().orElseThrow()); + Assertions.assertEquals(properties, provider.credentialToProperties(expected).orElseThrow()); + } + } + + @Test + void clearingImpersonationDoesNotAddProviderTypeToAlter() { + ObjectStorageAuthentication auth = provider.resolveAuthentication(Map.of( + "provider", "GCP", "gs.impersonation_service_account", "")).orElseThrow(); + Assertions.assertEquals(new GcpCredentialSpec(GcpCredentialSpec.ProviderType.DEFAULT, null), + auth.getCredential().orElseThrow()); + Assertions.assertEquals(new GcpCredentialSpec(null, ""), auth.getCredentialUpdates().orElseThrow()); + } + + @Test + void onlyGcpDescriptionsAreRestoredAndAbsentSourceUsesLegacyDefault() { + Assertions.assertEquals(Map.of("gs.credential_provider_type", "DEFAULT"), + provider.credentialToProperties(new GcpCredentialSpec(null, null)).orElseThrow()); + NativeObjectStorageCredential foreign = new NativeObjectStorageCredential() { + @Override + public String getProvider() { + return "OTHER"; + } + + @Override + public boolean isEmpty() { + return false; + } + }; + Assertions.assertTrue(provider.credentialToProperties(foreign).isEmpty()); + } + + @Test + void pluginCrossesTheSharedApiWithoutAccessToHostWireClasses() throws Exception { + URL[] urls = {GcsFileSystemProvider.class.getProtectionDomain().getCodeSource().getLocation(), + GcsAuthResolver.class.getProtectionDomain().getCodeSource().getLocation()}; + try (URLClassLoader loader = new URLClassLoader(urls, getClass().getClassLoader()) { + @Override + protected synchronized Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + if (name.startsWith("org.apache.doris.thrift.") || name.startsWith("org.apache.doris.cloud.proto.") + || name.startsWith("org.apache.doris.storage.protocol.")) { + throw new ClassNotFoundException("Protocol classes are unavailable to this plugin: " + name); + } + if (name.startsWith("org.apache.doris.filesystem.gcs.")) { + Class loaded = findLoadedClass(name); + if (loaded == null) { + loaded = findClass(name); + } + if (resolve) { + resolveClass(loaded); + } + return loaded; + } + return super.loadClass(name, resolve); + } + }) { + FileSystemProvider isolated = (FileSystemProvider) loader.loadClass( + GcsFileSystemProvider.class.getName()).getConstructor().newInstance(); + Assertions.assertNotSame(GcsFileSystemProvider.class, isolated.getClass()); + NativeObjectStorageCredential credential = isolated.resolveAuthentication( + Map.of("provider", "GCP")).orElseThrow().getCredential().orElseThrow(); + Assertions.assertEquals(GcpCredentialSpec.class, credential.getClass()); + Assertions.assertEquals(Map.of("gs.credential_provider_type", "DEFAULT"), + isolated.credentialToProperties(credential).orElseThrow()); + } + } + + @Test + void hmacAndAnonymousAuthenticationDoNotProduceNativeCredentials() { + for (Map properties : java.util.List.of( + Map.of("provider", "GCP", "s3.access_key", "ak", "s3.secret_key", "sk"), + Map.of("provider", "GCP", "gs.credential_provider_type", "ANONYMOUS"))) { + ObjectStorageAuthentication auth = provider.resolveAuthentication(properties).orElseThrow(); + Assertions.assertFalse(auth.isNative()); + Assertions.assertTrue(auth.getCredential().isEmpty()); + Assertions.assertTrue(auth.getCredentialUpdates().isEmpty()); + } + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java index 1d088e10bd786d..bc9a749c33cd5d 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java @@ -18,6 +18,7 @@ package org.apache.doris.filesystem.gcs; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.GcpCredentialSpec; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.s3.S3FileSystem; import org.apache.doris.filesystem.s3.S3ObjStorage; @@ -263,20 +264,21 @@ void nativeAuthenticationSeparatesCreateDefaultsFromExplicitAlterFields() { Assertions.assertEquals("GCP", defaults.getProvider()); Assertions.assertTrue(defaults.isNative()); Assertions.assertFalse(defaults.isAnonymous()); - Assertions.assertEquals(Map.of("credential_provider_type", "DEFAULT"), defaults.getCredential()); + Assertions.assertEquals(new GcpCredentialSpec(GcpCredentialSpec.ProviderType.DEFAULT, null), + defaults.getCredential().orElseThrow()); Assertions.assertTrue(defaults.getCredentialUpdates().isEmpty()); ObjectStorageAuthentication clear = provider.resolveAuthentication( Map.of("gs.impersonation_service_account", "")).get(); Assertions.assertEquals(defaults.getCredential(), clear.getCredential()); - Assertions.assertEquals(Map.of("impersonation_service_account", ""), clear.getCredentialUpdates()); - Assertions.assertFalse(clear.getCredentialUpdates().containsKey("credential_provider_type")); + Assertions.assertEquals(new GcpCredentialSpec(null, ""), clear.getCredentialUpdates().orElseThrow()); Map properties = Map.of("gs.credential_provider_type", "COMPUTE_ENGINE", "gs.impersonation_service_account", "reader@project.iam.gserviceaccount.com"); ObjectStorageAuthentication explicit = provider.resolveAuthentication(properties).get(); Assertions.assertEquals(explicit.getCredential(), explicit.getCredentialUpdates()); - Assertions.assertEquals(properties, provider.credentialToProperties(explicit.getCredential())); + Assertions.assertEquals(properties, + provider.credentialToProperties(explicit.getCredential().orElseThrow()).orElseThrow()); } @Test diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java index a143185d3f07aa..b4be3bfb1f6e57 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java @@ -20,6 +20,7 @@ import org.apache.doris.extension.spi.Plugin; import org.apache.doris.extension.spi.PluginFactory; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; import org.apache.doris.filesystem.auth.ObjectStorageAuthentication; import org.apache.doris.filesystem.properties.FileSystemCapability; import org.apache.doris.filesystem.properties.FileSystemProperties; @@ -78,16 +79,14 @@ default Set fallbackProviders() { return Collections.emptySet(); } - /** Optional codec for this provider's native BE/Cloud credentials. */ - default Optional credentialCodec() { + /** + * Restore provider-owned property names from a typed identity. Empty means this plugin does not + * handle the credential type. Neither user property names nor wire formats are interpreted by core. + */ + default Optional> credentialToProperties(NativeObjectStorageCredential credential) { return Optional.empty(); } - /** Translate protocol-native fields back into provider-owned user properties. */ - default Map credentialToProperties(Map credential) { - throw new UnsupportedOperationException(name() + " has no native credential property codec"); - } - /** Provider-owned credential properties accepted by a storage vault ALTER. */ default Set modifiableCredentialPropertyKeys() { return Collections.emptySet(); diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java deleted file mode 100644 index 1b14f2258298dc..00000000000000 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageCredentialCodec.java +++ /dev/null @@ -1,42 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.filesystem.spi; - -import java.io.IOException; -import java.util.Map; - -/** - * Provider-owned conversion between resolved credentials and the existing BE/Cloud wire envelopes. - * - *

Only JDK values cross the plugin boundary. Generated protocol classes and their runtimes remain - * implementation details; plugins must not depend on fe-core. Implementations perform no I/O and - * preserve absent fields versus explicitly empty fields in partial ALTER credentials.

- */ -public interface ObjectStorageCredentialCodec { - /** Canonical object-store provider name used by the BE and Cloud protocols. */ - String provider(); - - /** Encode a complete ObjectStoreCredentialPB envelope, not its provider-specific submessage. */ - byte[] toCloudCredential(Map credential) throws IOException; - - /** Encode a complete TCredential envelope using Thrift's binary protocol. */ - byte[] toThriftCredential(Map credential) throws IOException; - - /** Restore user-facing properties from an ObjectStoreCredentialPB envelope. */ - Map toProperties(byte[] cloudCredential) throws IOException; -} diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java index dd6644e9f7890e..ba8f429846c27d 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemPluginSurfaceTest.java @@ -68,7 +68,9 @@ public class FileSystemPluginSurfaceTest { /** The types a filesystem plugin implements or calls. Everything reachable on them is the contract. */ private static final List> FROZEN_TYPES = Arrays.asList( FileSystemProvider.class, - ObjectStorageCredentialCodec.class, + org.apache.doris.filesystem.auth.NativeObjectStorageCredential.class, + org.apache.doris.filesystem.auth.GcpCredentialSpec.class, + org.apache.doris.filesystem.auth.GcpCredentialSpec.ProviderType.class, org.apache.doris.filesystem.auth.ObjectStorageAuthentication.class, ObjFileSystem.class, ObjStorage.class, diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java index b1aa754e002350..a63fc103000012 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/java/org/apache/doris/filesystem/spi/FileSystemProviderTypedPropertiesTest.java @@ -47,8 +47,9 @@ void provider_canBindAndCreateWithTypedProperties() throws IOException { } @Test - void providerWithoutNativeCredentialsDoesNotAdvertiseACodec() { - Assertions.assertTrue(new TypedProvider().credentialCodec().isEmpty()); + void providerWithoutNativeCredentialsDoesNotClaimCredentialProperties() { + Assertions.assertTrue(new TypedProvider().credentialToProperties( + new org.apache.doris.filesystem.auth.GcpCredentialSpec(null, null)).isEmpty()); } private static class TypedProvider implements FileSystemProvider { diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt index d06e058591c2f1..0cda5a09b8212c 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt @@ -5,11 +5,33 @@ org.apache.doris.extension.spi.PluginFactory#create():org.apache.doris.extension org.apache.doris.extension.spi.PluginFactory#create(org.apache.doris.extension.spi.PluginContext):org.apache.doris.extension.spi.Plugin org.apache.doris.extension.spi.PluginFactory#description():java.lang.String org.apache.doris.extension.spi.PluginFactory#name():java.lang.String -org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getCredential():java.util.Map -org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getCredentialUpdates():java.util.Map +org.apache.doris.filesystem.auth.GcpCredentialSpec#equals(java.lang.Object):boolean +org.apache.doris.filesystem.auth.GcpCredentialSpec#getImpersonationServiceAccount():java.util.Optional +org.apache.doris.filesystem.auth.GcpCredentialSpec#getProvider():java.lang.String +org.apache.doris.filesystem.auth.GcpCredentialSpec#getProviderType():java.util.Optional +org.apache.doris.filesystem.auth.GcpCredentialSpec#hashCode():int +org.apache.doris.filesystem.auth.GcpCredentialSpec#isEmpty():boolean +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#compareTo(java.lang.Enum):int +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#describeConstable():java.util.Optional +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#equals(java.lang.Object):boolean +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#getDeclaringClass():java.lang.Class +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#hashCode():int +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#name():java.lang.String +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#ordinal():int +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#toString():java.lang.String +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#valueOf(java.lang.Class,java.lang.String):java.lang.Enum +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#valueOf(java.lang.String):org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType +org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType#values():org.apache.doris.filesystem.auth.GcpCredentialSpec$ProviderType[] +org.apache.doris.filesystem.auth.NativeObjectStorageCredential#getProvider():java.lang.String +org.apache.doris.filesystem.auth.NativeObjectStorageCredential#isEmpty():boolean +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#anonymous(java.lang.String):org.apache.doris.filesystem.auth.ObjectStorageAuthentication +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getCredential():java.util.Optional +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getCredentialUpdates():java.util.Optional org.apache.doris.filesystem.auth.ObjectStorageAuthentication#getProvider():java.lang.String +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#hmac(java.lang.String):org.apache.doris.filesystem.auth.ObjectStorageAuthentication org.apache.doris.filesystem.auth.ObjectStorageAuthentication#isAnonymous():boolean org.apache.doris.filesystem.auth.ObjectStorageAuthentication#isNative():boolean +org.apache.doris.filesystem.auth.ObjectStorageAuthentication#nativeAuthentication(org.apache.doris.filesystem.auth.NativeObjectStorageCredential,org.apache.doris.filesystem.auth.NativeObjectStorageCredential):org.apache.doris.filesystem.auth.ObjectStorageAuthentication org.apache.doris.filesystem.spi.FileSystemProvider#bind(java.util.Map):org.apache.doris.filesystem.properties.FileSystemProperties org.apache.doris.filesystem.spi.FileSystemProvider#capabilities(org.apache.doris.filesystem.properties.FileSystemProperties):java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#clearablePropertyKeys():java.util.Set @@ -18,8 +40,7 @@ org.apache.doris.filesystem.spi.FileSystemProvider#create(java.util.Map):org.apa org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.extension.spi.PluginContext):org.apache.doris.extension.spi.Plugin org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem org.apache.doris.filesystem.spi.FileSystemProvider#createUntyped(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem -org.apache.doris.filesystem.spi.FileSystemProvider#credentialCodec():java.util.Optional -org.apache.doris.filesystem.spi.FileSystemProvider#credentialToProperties(java.util.Map):java.util.Map +org.apache.doris.filesystem.spi.FileSystemProvider#credentialToProperties(org.apache.doris.filesystem.auth.NativeObjectStorageCredential):java.util.Optional org.apache.doris.filesystem.spi.FileSystemProvider#description():java.lang.String org.apache.doris.filesystem.spi.FileSystemProvider#fallbackProviders():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#modifiableCredentialPropertyKeys():java.util.Set @@ -75,7 +96,3 @@ org.apache.doris.filesystem.spi.ObjStorage#listObjectsWithPrefix(java.lang.Strin org.apache.doris.filesystem.spi.ObjStorage#openInputStreamAt(java.lang.String,long):java.io.InputStream org.apache.doris.filesystem.spi.ObjStorage#putObject(java.lang.String,org.apache.doris.filesystem.spi.RequestBody):void org.apache.doris.filesystem.spi.ObjStorage#uploadPart(java.lang.String,java.lang.String,int,org.apache.doris.filesystem.spi.RequestBody):org.apache.doris.filesystem.UploadPartResult -org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#provider():java.lang.String -org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#toCloudCredential(java.util.Map):byte[] -org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#toProperties(byte[]):java.util.Map -org.apache.doris.filesystem.spi.ObjectStorageCredentialCodec#toThriftCredential(java.util.Map):byte[] diff --git a/fe/fe-filesystem/pom.xml b/fe/fe-filesystem/pom.xml index 1d6dd09d27f612..2ff20328f23c54 100644 --- a/fe/fe-filesystem/pom.xml +++ b/fe/fe-filesystem/pom.xml @@ -53,7 +53,7 @@ under the License. bumping all five families. See plan-doc/designs/2026-07-29-plugin-api-version-check-design.md. --> - 4.0 + 5.0 diff --git a/fe/fe-storage-protocol/pom.xml b/fe/fe-storage-protocol/pom.xml new file mode 100644 index 00000000000000..9b8ed9504f7809 --- /dev/null +++ b/fe/fe-storage-protocol/pom.xml @@ -0,0 +1,55 @@ + + + + 4.0.0 + + org.apache.doris + fe + ${revision} + ../pom.xml + + fe-storage-protocol + Doris object storage protocol adapters + Host-side translation of typed filesystem identities into Doris RPC messages. + + + org.apache.doris + fe-filesystem-api + ${revision} + + + org.apache.doris + fe-grpc + ${revision} + + + org.apache.doris + fe-thrift + ${revision} + + + org.junit.jupiter + junit-jupiter + test + + + diff --git a/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/GcpCredentialCodec.java b/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/GcpCredentialCodec.java new file mode 100644 index 00000000000000..be06993bd00432 --- /dev/null +++ b/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/GcpCredentialCodec.java @@ -0,0 +1,86 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.storage.protocol; + +import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.filesystem.auth.GcpCredentialSpec; +import org.apache.doris.thrift.TCredential; +import org.apache.doris.thrift.TGcpCredential; +import org.apache.doris.thrift.TGcpCredentialProviderType; + +/** GCP wire-schema mapping only; property validation and credential loading stay in the GCS plugin. */ +final class GcpCredentialCodec implements ObjectStorageCredentialCodec { + @Override + public Class credentialType() { + return GcpCredentialSpec.class; + } + + @Override + public ObjectStoreCredentialPB toCloud(GcpCredentialSpec spec) { + GcpCredentialPB.Builder credential = GcpCredentialPB.newBuilder(); + spec.getProviderType().ifPresent(type -> { + GcpCredentialPB.CredentialProviderType mapped = switch (type) { + case DEFAULT -> GcpCredentialPB.CredentialProviderType.DEFAULT; + case COMPUTE_ENGINE -> GcpCredentialPB.CredentialProviderType.COMPUTE_ENGINE; + default -> throw new IllegalArgumentException("Unsupported GCP identity source: " + type); + }; + credential.setCredentialProviderType(mapped); + }); + spec.getImpersonationServiceAccount().ifPresent(credential::setImpersonationServiceAccount); + return ObjectStoreCredentialPB.newBuilder().setGcpCredential(credential).build(); + } + + @Override + public TCredential toThrift(GcpCredentialSpec spec) { + TGcpCredential credential = new TGcpCredential(); + spec.getProviderType().ifPresent(type -> { + TGcpCredentialProviderType mapped = switch (type) { + case DEFAULT -> TGcpCredentialProviderType.DEFAULT; + case COMPUTE_ENGINE -> TGcpCredentialProviderType.COMPUTE_ENGINE; + default -> throw new IllegalArgumentException("Unsupported GCP identity source: " + type); + }; + credential.setCredentialProviderType(mapped); + }); + spec.getImpersonationServiceAccount().ifPresent(credential::setImpersonationServiceAccount); + return new TCredential().setGcpCredential(credential); + } + + @Override + public GcpCredentialSpec fromCloud(ObjectStoreCredentialPB envelope) { + if (!envelope.hasGcpCredential()) { + throw new IllegalArgumentException("Missing GCP credentials in the Cloud credential envelope"); + } + GcpCredentialPB credential = envelope.getGcpCredential(); + // Proto2 retains unknown enum values as unknown fields. Treating that as absence would + // make a newer identity source silently fall back to ADC when properties are restored. + if (credential.getUnknownFields().hasField(GcpCredentialPB.CREDENTIAL_PROVIDER_TYPE_FIELD_NUMBER)) { + throw new IllegalArgumentException("Unsupported GCP identity source in Cloud credentials"); + } + GcpCredentialSpec.ProviderType type = null; + if (credential.hasCredentialProviderType()) { + type = switch (credential.getCredentialProviderType()) { + case DEFAULT -> GcpCredentialSpec.ProviderType.DEFAULT; + case COMPUTE_ENGINE -> GcpCredentialSpec.ProviderType.COMPUTE_ENGINE; + default -> throw new IllegalArgumentException("Unsupported GCP identity source in Cloud credentials"); + }; + } + return new GcpCredentialSpec(type, + credential.hasImpersonationServiceAccount() ? credential.getImpersonationServiceAccount() : null); + } +} diff --git a/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialCodec.java b/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialCodec.java new file mode 100644 index 00000000000000..cd9d19062d7e8c --- /dev/null +++ b/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialCodec.java @@ -0,0 +1,33 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.storage.protocol; + +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; +import org.apache.doris.thrift.TCredential; + +/** Internal host adapter contract; never implemented or loaded by filesystem plugins. */ +interface ObjectStorageCredentialCodec { + Class credentialType(); + + ObjectStoreCredentialPB toCloud(C credential); + + TCredential toThrift(C credential); + + C fromCloud(ObjectStoreCredentialPB credential); +} diff --git a/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslator.java b/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslator.java new file mode 100644 index 00000000000000..975afbd0022147 --- /dev/null +++ b/fe/fe-storage-protocol/src/main/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslator.java @@ -0,0 +1,76 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.storage.protocol; + +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; +import org.apache.doris.thrift.TCredential; + +import java.util.Locale; +import java.util.Map; + +/** + * Host-side boundary between identity descriptions and the wire variants understood by Doris. + * Adding a wire variant requires a codec here and matching receiver support, not a core factory branch + * or a serialization hook in the filesystem SPI. No user properties or SDK objects enter this module. + */ +public final class ObjectStorageCredentialTranslator { + private static final Map> CODECS = Map.of("GCP", new GcpCredentialCodec()); + + private ObjectStorageCredentialTranslator() { + } + + public static ObjectStoreCredentialPB toCloud(NativeObjectStorageCredential credential) { + return toCloud(codec(credential.getProvider()), credential); + } + + public static TCredential toThrift(NativeObjectStorageCredential credential) { + return toThrift(codec(credential.getProvider()), credential); + } + + public static NativeObjectStorageCredential fromCloud(String provider, ObjectStoreCredentialPB credential) { + return codec(provider).fromCloud(credential); + } + + private static ObjectStorageCredentialCodec codec(String provider) { + ObjectStorageCredentialCodec codec = CODECS.get(provider.toUpperCase(Locale.ROOT)); + if (codec == null) { + throw new IllegalArgumentException("Unsupported native credential protocol for provider: " + provider); + } + return codec; + } + + private static C checkedCredential( + ObjectStorageCredentialCodec codec, NativeObjectStorageCredential credential) { + if (!codec.credentialType().isInstance(credential)) { + throw new IllegalArgumentException("Unexpected credential description for provider: " + + credential.getProvider()); + } + return codec.credentialType().cast(credential); + } + + private static ObjectStoreCredentialPB toCloud( + ObjectStorageCredentialCodec codec, NativeObjectStorageCredential credential) { + return codec.toCloud(checkedCredential(codec, credential)); + } + + private static TCredential toThrift( + ObjectStorageCredentialCodec codec, NativeObjectStorageCredential credential) { + return codec.toThrift(checkedCredential(codec, credential)); + } +} diff --git a/fe/fe-storage-protocol/src/test/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslatorTest.java b/fe/fe-storage-protocol/src/test/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslatorTest.java new file mode 100644 index 00000000000000..c2046c5685e1eb --- /dev/null +++ b/fe/fe-storage-protocol/src/test/java/org/apache/doris/storage/protocol/ObjectStorageCredentialTranslatorTest.java @@ -0,0 +1,116 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.storage.protocol; + +import org.apache.doris.cloud.proto.Cloud.GcpCredentialPB; +import org.apache.doris.cloud.proto.Cloud.ObjectStoreCredentialPB; +import org.apache.doris.filesystem.auth.GcpCredentialSpec; +import org.apache.doris.filesystem.auth.NativeObjectStorageCredential; +import org.apache.doris.thrift.TGcpCredential; +import org.apache.doris.thrift.TGcpCredentialProviderType; + +import com.google.protobuf.UnknownFieldSet; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +class ObjectStorageCredentialTranslatorTest { + @Test + void bothWireFormatsPreserveAllSupportedIdentitySources() { + for (GcpCredentialSpec.ProviderType type : GcpCredentialSpec.ProviderType.values()) { + GcpCredentialSpec spec = new GcpCredentialSpec(type, "target@test.iam.gserviceaccount.com"); + ObjectStoreCredentialPB cloud = ObjectStorageCredentialTranslator.toCloud(spec); + TGcpCredential thrift = ObjectStorageCredentialTranslator.toThrift(spec).getGcpCredential(); + Assertions.assertEquals(GcpCredentialPB.CredentialProviderType.valueOf(type.name()), + cloud.getGcpCredential().getCredentialProviderType()); + Assertions.assertEquals(TGcpCredentialProviderType.valueOf(type.name()), thrift.getCredentialProviderType()); + Assertions.assertEquals("target@test.iam.gserviceaccount.com", thrift.getImpersonationServiceAccount()); + Assertions.assertEquals(thrift.getImpersonationServiceAccount(), + cloud.getGcpCredential().getImpersonationServiceAccount()); + Assertions.assertEquals(spec, ObjectStorageCredentialTranslator.fromCloud("gcp", cloud)); + } + } + + @Test + void partialAlterPreservesExplicitClearWithoutAddingSource() { + GcpCredentialSpec patch = new GcpCredentialSpec(null, ""); + ObjectStoreCredentialPB cloud = ObjectStorageCredentialTranslator.toCloud(patch); + TGcpCredential thrift = ObjectStorageCredentialTranslator.toThrift(patch).getGcpCredential(); + Assertions.assertFalse(cloud.getGcpCredential().hasCredentialProviderType()); + Assertions.assertFalse(thrift.isSetCredentialProviderType()); + Assertions.assertTrue(cloud.getGcpCredential().hasImpersonationServiceAccount()); + Assertions.assertTrue(thrift.isSetImpersonationServiceAccount()); + Assertions.assertEquals("", thrift.getImpersonationServiceAccount()); + Assertions.assertEquals("", cloud.getGcpCredential().getImpersonationServiceAccount()); + Assertions.assertEquals(patch, ObjectStorageCredentialTranslator.fromCloud("GCP", cloud)); + } + + @Test + void absentFieldsStayAbsentInBothDirections() { + for (GcpCredentialSpec spec : new GcpCredentialSpec[] { + new GcpCredentialSpec(GcpCredentialSpec.ProviderType.DEFAULT, null), + new GcpCredentialSpec(null, null)}) { + ObjectStoreCredentialPB cloud = ObjectStorageCredentialTranslator.toCloud(spec); + TGcpCredential thrift = ObjectStorageCredentialTranslator.toThrift(spec).getGcpCredential(); + Assertions.assertFalse(cloud.getGcpCredential().hasImpersonationServiceAccount()); + Assertions.assertFalse(thrift.isSetImpersonationServiceAccount()); + Assertions.assertEquals(spec, ObjectStorageCredentialTranslator.fromCloud("GCP", cloud)); + Assertions.assertEquals(spec.getProviderType().isPresent(), thrift.isSetCredentialProviderType()); + } + } + + @Test + void missingEnvelopeDoesNotSilentlySelectDefaultCredentials() { + Assertions.assertThrows(IllegalArgumentException.class, () -> ObjectStorageCredentialTranslator.fromCloud( + "GCP", ObjectStoreCredentialPB.getDefaultInstance())); + } + + @Test + void unknownFutureSourceDoesNotTurnIntoApplicationDefaultCredentials() throws Exception { + GcpCredentialPB unknown = GcpCredentialPB.newBuilder().setUnknownFields(UnknownFieldSet.newBuilder() + .addField(GcpCredentialPB.CREDENTIAL_PROVIDER_TYPE_FIELD_NUMBER, + UnknownFieldSet.Field.newBuilder().addVarint(100).build()).build()).build(); + ObjectStoreCredentialPB envelope = ObjectStoreCredentialPB.parseFrom( + ObjectStoreCredentialPB.newBuilder().setGcpCredential(unknown).build().toByteArray()); + Assertions.assertFalse(envelope.getGcpCredential().hasCredentialProviderType()); + Assertions.assertThrows(IllegalArgumentException.class, + () -> ObjectStorageCredentialTranslator.fromCloud("GCP", envelope)); + } + + @Test + void unsupportedProvidersAndMismatchedTypesFailExplicitly() { + for (String provider : new String[] {"OTHER", "GCP"}) { + NativeObjectStorageCredential credential = new NativeObjectStorageCredential() { + @Override + public String getProvider() { + return provider; + } + + @Override + public boolean isEmpty() { + return false; + } + }; + Assertions.assertThrows(IllegalArgumentException.class, + () -> ObjectStorageCredentialTranslator.toCloud(credential)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> ObjectStorageCredentialTranslator.toThrift(credential)); + } + Assertions.assertThrows(IllegalArgumentException.class, () -> ObjectStorageCredentialTranslator.fromCloud( + "OTHER", ObjectStoreCredentialPB.getDefaultInstance())); + } +} diff --git a/fe/pom.xml b/fe/pom.xml index e5a4185e330445..ae0c86b7db80cc 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -233,6 +233,7 @@ under the License. fe-iceberg-common fe-catalog fe-filesystem + fe-storage-protocol fe-extension-spi fe-connector fe-extension-loader From 4757dcc82a21f1791304f42189cffd0c90f00529 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 19 Oct 2026 14:23:15 +0800 Subject: [PATCH 32/47] [fix](test) Align CI fixtures with current Variant and URI semantics ### What problem does this PR solve? Related PR: #68540 Problem Summary: BE unit tests still initialize the removed check_duplicate_json_path option after the master merge. Regression fixtures also depend on implicit invalid-JSON string parsing, the old missing-scheme error message, and an INT96 timezone that shifts the HTTP fixture by eight hours. Remove the obsolete initializers, use explicit JSON string documents, check that an invalid Paimon insert fails before a valid insert succeeds, expect the current URI validation message, and read the INT96 fixture in UTC. Keep the existing generated result files unchanged. ### Release note None ### Check List (For Author) - Test: clang-format 16 checks on all changed C++ files, Groovy 4 parsing of all four changed suites, BE build hygiene, and git diff --check passed. Standard BE UT and regression runners were attempted but could not start without a usable JDK 17. No local Doris/Paimon test cluster is available; full runtime validation remains pending CI. - Behavior changed: No, test fixtures and assertions only. - Does this need documentation: No --- .../data_type_serde/data_type_serde_arrow_test.cpp | 10 ++++------ .../data_type_variant_v2_serde_jsonb_test.cpp | 5 ++--- be/test/exprs/variant_v2_execution_test.cpp | 5 ++--- .../write/test_paimon_write_variant_errors.groovy | 14 +++++++++++--- .../external_table_p0/tvf/test_http_tvf.groovy | 10 +++++----- .../test_streaming_insert_job_crud.groovy | 2 +- .../variant_p0/test_variant_json_functions.groovy | 2 +- 7 files changed, 26 insertions(+), 22 deletions(-) diff --git a/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp b/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp index c32d552788de97..902a55bf1851be 100644 --- a/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp +++ b/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp @@ -957,9 +957,8 @@ TEST(DataTypeSerDeArrowTest, PlainArrowConverterDoesNotInferIcebergUuid) { TEST(DataTypeSerDeArrowTest, IcebergVariantExtensionAndParquetSchema) { auto make_variant_column = []() { - JsonStringToVariantEncoder encoder({.max_json_key_length = 1024, - .throw_on_invalid_json = true, - .check_duplicate_json_path = false}); + JsonStringToVariantEncoder encoder( + {.max_json_key_length = 1024, .throw_on_invalid_json = true}); for (std::string_view json : {std::string_view {R"({"name":"doris","n":1})"}, std::string_view {"null"}, std::string_view {R"([1,true,"x"])"}}) { @@ -1054,9 +1053,8 @@ TEST(DataTypeSerDeArrowTest, IcebergVariantExtensionAndParquetSchema) { TEST(DataTypeSerDeArrowTest, NestedIcebergVariantExtensionsAndParquetSchema) { auto make_variant_column = [](std::initializer_list json_values) { - JsonStringToVariantEncoder encoder({.max_json_key_length = 1024, - .throw_on_invalid_json = true, - .check_duplicate_json_path = false}); + JsonStringToVariantEncoder encoder( + {.max_json_key_length = 1024, .throw_on_invalid_json = true}); for (std::string_view json : json_values) { encoder.add_json({json.data(), json.size()}); } diff --git a/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp b/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp index 5cecf9d9cefa00..a88aeec2d48e42 100644 --- a/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp +++ b/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp @@ -112,9 +112,8 @@ std::string json_at(const DataTypeVariantV2SerDe& serde, const IColumn& column, } ColumnVariantV2::MutablePtr encoded_json(std::initializer_list rows) { - JsonStringToVariantEncoder encoder({.max_json_key_length = 1024, - .throw_on_invalid_json = true, - .check_duplicate_json_path = false}); + JsonStringToVariantEncoder encoder( + {.max_json_key_length = 1024, .throw_on_invalid_json = true}); for (std::string_view row : rows) { encoder.add_json({row.data(), row.size()}); } diff --git a/be/test/exprs/variant_v2_execution_test.cpp b/be/test/exprs/variant_v2_execution_test.cpp index e3db965fc222cb..75d045cf4b4e7b 100644 --- a/be/test/exprs/variant_v2_execution_test.cpp +++ b/be/test/exprs/variant_v2_execution_test.cpp @@ -49,9 +49,8 @@ TEST(VariantV2ExecutionTest, TypePropertiesDoNotAffectEquality) { TEST(VariantV2ExecutionTest, CanonicalComparison) { auto parse = [](std::initializer_list rows) { - JsonStringToVariantEncoder encoder({.max_json_key_length = 255, - .throw_on_invalid_json = true, - .check_duplicate_json_path = false}); + JsonStringToVariantEncoder encoder( + {.max_json_key_length = 255, .throw_on_invalid_json = true}); for (auto json : rows) { encoder.add_json({json.data(), json.size()}); } diff --git a/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy b/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy index 0ee47c376440c9..0ab160bdaccf6a 100644 --- a/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy +++ b/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy @@ -129,10 +129,19 @@ suite("test_paimon_write_variant_errors", "p0,external,paimon,nonConcurrent") { LATERAL VIEW explode(ARRAY(IF(TRUE, payload, 1))) generated AS generated_payload """ + // Invalid JSON now fails parsing instead of becoming a Variant string. Verify that + // the failed insert leaves the sink usable before writing the valid rows below. + test { + sql """ + INSERT INTO t_variant_error VALUES (20, parse_to_variant('not-json')) + """ + exception "Parse json document failed at row 0, error:" + } + sql """ INSERT INTO t_variant_error VALUES (20, parse_to_variant('{"recovered":true}')), - (21, try_parse_to_variant('not-json')), + (21, parse_to_variant('"not-json"')), (22, CAST(CAST('{"typed":"string"}' AS STRING) AS VARIANT)) """ // Primitive-to-Variant coercion remains supported for both inline VALUES and ordinary SELECT. @@ -190,8 +199,7 @@ suite("test_paimon_write_variant_errors", "p0,external,paimon,nonConcurrent") { row.collect { value -> value == null ? null : value.toString() } }) - // Invalid JSON is preserved as a Variant string unless the global - // throw-on-invalid-JSON option is enabled. + // The recovered object and the explicitly quoted JSON string survive the failed insert. order_qt_variant_after_errors """ SELECT id, payload IS NULL, CAST(payload['recovered'] AS BOOLEAN), diff --git a/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy b/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy index 485d267e48a475..0188877b6fa78f 100644 --- a/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy +++ b/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy @@ -239,14 +239,14 @@ suite("test_http_tvf", "p0") { ); """ - // The Spark INT96 fixture stores timestamps normalized to UTC; the golden uses Asia/Shanghai. - // Specify the INT96 timezone explicitly instead of relying on the reader's default semantics. + // Preserve the fixture's INT96 wall-clock values; applying Asia/Shanghai adds eight hours. + // Specify UTC explicitly so the result does not depend on the reader's default timezone. qt_sql08 """ select * from http( "uri" = "${httpUrl("external_table_p0/tvf/t.parquet")}", "format" = "parquet", - "hive.parquet.time-zone" = "Asia/Shanghai" + "hive.parquet.time-zone" = "UTC" ) order by id limit 10; """ @@ -255,7 +255,7 @@ suite("test_http_tvf", "p0") { http( "uri" = "${httpUrl("external_table_p0/tvf/t.parquet")}", "format" = "parquet", - "hive.parquet.time-zone" = "Asia/Shanghai" + "hive.parquet.time-zone" = "UTC" ) order by id limit 10; """ @@ -264,7 +264,7 @@ suite("test_http_tvf", "p0") { http( "uri" = "${httpUrl("external_table_p0/tvf/t.parquet")}", "format" = "parquet", - "hive.parquet.time-zone" = "Asia/Shanghai" + "hive.parquet.time-zone" = "UTC" ); """ diff --git a/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy b/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy index 8de7f1b2748b92..732c28f41c7a5c 100644 --- a/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy +++ b/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy @@ -231,7 +231,7 @@ suite("test_streaming_insert_job_crud") { "s3.secret_key" = "${getS3SK()}" ); """ - }, "Invalid uri") + }, "Invalid scheme") jobCount = sql """ select count(1) from jobs("type"="insert") where Name = '${jobNameError}' and ExecuteType='STREAMING' """ assert jobCount.get(0).get(0) == 0 diff --git a/regression-test/suites/variant_p0/test_variant_json_functions.groovy b/regression-test/suites/variant_p0/test_variant_json_functions.groovy index 458733170301a2..007a94a0aae004 100644 --- a/regression-test/suites/variant_p0/test_variant_json_functions.groovy +++ b/regression-test/suites/variant_p0/test_variant_json_functions.groovy @@ -30,7 +30,7 @@ suite("test_variant_json_functions", "p0") { UNION ALL SELECT 2, '{"a":2,"s":"yz","b":{"c":[3]},"n":{"m":true}}' UNION ALL SELECT 3, '[1,"two",{"three":3}]' UNION ALL SELECT 4, '42' - UNION ALL SELECT 5, 'abc' + UNION ALL SELECT 5, '"abc"' UNION ALL SELECT 6, NULL) t """ From f187fb2827fd2a244587b63cbbb3681521cb6234 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Mon, 19 Oct 2026 14:34:42 +0800 Subject: [PATCH 33/47] [fix](test) Revert unrelated CI fixture fixes from GCP IAM PR ### What problem does this PR solve? Related PR: #68540 Problem Summary: Revert commit 4757dcc82a21f1791304f42189cffd0c90f00529. Its test fixes address issues inherited from master and are outside the native GCP IAM change. Restore the exact source tree preceding those test fixes. ### Release note None ### Check List (For Author) - Test: git diff --cached --check passed; verified the restored tree matches 80d7faecaee0ab7831e08d3fef3b0ff86221fa89. Runtime tests not rerun for this exact revert. - Behavior changed: No change to the original GCP IAM PR. - Does this need documentation: No --- .../data_type_serde/data_type_serde_arrow_test.cpp | 10 ++++++---- .../data_type_variant_v2_serde_jsonb_test.cpp | 5 +++-- be/test/exprs/variant_v2_execution_test.cpp | 5 +++-- .../write/test_paimon_write_variant_errors.groovy | 14 +++----------- .../external_table_p0/tvf/test_http_tvf.groovy | 10 +++++----- .../test_streaming_insert_job_crud.groovy | 2 +- .../variant_p0/test_variant_json_functions.groovy | 2 +- 7 files changed, 22 insertions(+), 26 deletions(-) diff --git a/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp b/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp index 902a55bf1851be..c32d552788de97 100644 --- a/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp +++ b/be/test/core/data_type_serde/data_type_serde_arrow_test.cpp @@ -957,8 +957,9 @@ TEST(DataTypeSerDeArrowTest, PlainArrowConverterDoesNotInferIcebergUuid) { TEST(DataTypeSerDeArrowTest, IcebergVariantExtensionAndParquetSchema) { auto make_variant_column = []() { - JsonStringToVariantEncoder encoder( - {.max_json_key_length = 1024, .throw_on_invalid_json = true}); + JsonStringToVariantEncoder encoder({.max_json_key_length = 1024, + .throw_on_invalid_json = true, + .check_duplicate_json_path = false}); for (std::string_view json : {std::string_view {R"({"name":"doris","n":1})"}, std::string_view {"null"}, std::string_view {R"([1,true,"x"])"}}) { @@ -1053,8 +1054,9 @@ TEST(DataTypeSerDeArrowTest, IcebergVariantExtensionAndParquetSchema) { TEST(DataTypeSerDeArrowTest, NestedIcebergVariantExtensionsAndParquetSchema) { auto make_variant_column = [](std::initializer_list json_values) { - JsonStringToVariantEncoder encoder( - {.max_json_key_length = 1024, .throw_on_invalid_json = true}); + JsonStringToVariantEncoder encoder({.max_json_key_length = 1024, + .throw_on_invalid_json = true, + .check_duplicate_json_path = false}); for (std::string_view json : json_values) { encoder.add_json({json.data(), json.size()}); } diff --git a/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp b/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp index a88aeec2d48e42..5cecf9d9cefa00 100644 --- a/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp +++ b/be/test/core/data_type_serde/data_type_variant_v2_serde_jsonb_test.cpp @@ -112,8 +112,9 @@ std::string json_at(const DataTypeVariantV2SerDe& serde, const IColumn& column, } ColumnVariantV2::MutablePtr encoded_json(std::initializer_list rows) { - JsonStringToVariantEncoder encoder( - {.max_json_key_length = 1024, .throw_on_invalid_json = true}); + JsonStringToVariantEncoder encoder({.max_json_key_length = 1024, + .throw_on_invalid_json = true, + .check_duplicate_json_path = false}); for (std::string_view row : rows) { encoder.add_json({row.data(), row.size()}); } diff --git a/be/test/exprs/variant_v2_execution_test.cpp b/be/test/exprs/variant_v2_execution_test.cpp index 75d045cf4b4e7b..e3db965fc222cb 100644 --- a/be/test/exprs/variant_v2_execution_test.cpp +++ b/be/test/exprs/variant_v2_execution_test.cpp @@ -49,8 +49,9 @@ TEST(VariantV2ExecutionTest, TypePropertiesDoNotAffectEquality) { TEST(VariantV2ExecutionTest, CanonicalComparison) { auto parse = [](std::initializer_list rows) { - JsonStringToVariantEncoder encoder( - {.max_json_key_length = 255, .throw_on_invalid_json = true}); + JsonStringToVariantEncoder encoder({.max_json_key_length = 255, + .throw_on_invalid_json = true, + .check_duplicate_json_path = false}); for (auto json : rows) { encoder.add_json({json.data(), json.size()}); } diff --git a/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy b/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy index 0ab160bdaccf6a..0ee47c376440c9 100644 --- a/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy +++ b/regression-test/suites/external_table_p0/paimon/write/test_paimon_write_variant_errors.groovy @@ -129,19 +129,10 @@ suite("test_paimon_write_variant_errors", "p0,external,paimon,nonConcurrent") { LATERAL VIEW explode(ARRAY(IF(TRUE, payload, 1))) generated AS generated_payload """ - // Invalid JSON now fails parsing instead of becoming a Variant string. Verify that - // the failed insert leaves the sink usable before writing the valid rows below. - test { - sql """ - INSERT INTO t_variant_error VALUES (20, parse_to_variant('not-json')) - """ - exception "Parse json document failed at row 0, error:" - } - sql """ INSERT INTO t_variant_error VALUES (20, parse_to_variant('{"recovered":true}')), - (21, parse_to_variant('"not-json"')), + (21, try_parse_to_variant('not-json')), (22, CAST(CAST('{"typed":"string"}' AS STRING) AS VARIANT)) """ // Primitive-to-Variant coercion remains supported for both inline VALUES and ordinary SELECT. @@ -199,7 +190,8 @@ suite("test_paimon_write_variant_errors", "p0,external,paimon,nonConcurrent") { row.collect { value -> value == null ? null : value.toString() } }) - // The recovered object and the explicitly quoted JSON string survive the failed insert. + // Invalid JSON is preserved as a Variant string unless the global + // throw-on-invalid-JSON option is enabled. order_qt_variant_after_errors """ SELECT id, payload IS NULL, CAST(payload['recovered'] AS BOOLEAN), diff --git a/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy b/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy index 0188877b6fa78f..485d267e48a475 100644 --- a/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy +++ b/regression-test/suites/external_table_p0/tvf/test_http_tvf.groovy @@ -239,14 +239,14 @@ suite("test_http_tvf", "p0") { ); """ - // Preserve the fixture's INT96 wall-clock values; applying Asia/Shanghai adds eight hours. - // Specify UTC explicitly so the result does not depend on the reader's default timezone. + // The Spark INT96 fixture stores timestamps normalized to UTC; the golden uses Asia/Shanghai. + // Specify the INT96 timezone explicitly instead of relying on the reader's default semantics. qt_sql08 """ select * from http( "uri" = "${httpUrl("external_table_p0/tvf/t.parquet")}", "format" = "parquet", - "hive.parquet.time-zone" = "UTC" + "hive.parquet.time-zone" = "Asia/Shanghai" ) order by id limit 10; """ @@ -255,7 +255,7 @@ suite("test_http_tvf", "p0") { http( "uri" = "${httpUrl("external_table_p0/tvf/t.parquet")}", "format" = "parquet", - "hive.parquet.time-zone" = "UTC" + "hive.parquet.time-zone" = "Asia/Shanghai" ) order by id limit 10; """ @@ -264,7 +264,7 @@ suite("test_http_tvf", "p0") { http( "uri" = "${httpUrl("external_table_p0/tvf/t.parquet")}", "format" = "parquet", - "hive.parquet.time-zone" = "UTC" + "hive.parquet.time-zone" = "Asia/Shanghai" ); """ diff --git a/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy b/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy index 732c28f41c7a5c..8de7f1b2748b92 100644 --- a/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy +++ b/regression-test/suites/job_p0/streaming_job/test_streaming_insert_job_crud.groovy @@ -231,7 +231,7 @@ suite("test_streaming_insert_job_crud") { "s3.secret_key" = "${getS3SK()}" ); """ - }, "Invalid scheme") + }, "Invalid uri") jobCount = sql """ select count(1) from jobs("type"="insert") where Name = '${jobNameError}' and ExecuteType='STREAMING' """ assert jobCount.get(0).get(0) == 0 diff --git a/regression-test/suites/variant_p0/test_variant_json_functions.groovy b/regression-test/suites/variant_p0/test_variant_json_functions.groovy index 007a94a0aae004..458733170301a2 100644 --- a/regression-test/suites/variant_p0/test_variant_json_functions.groovy +++ b/regression-test/suites/variant_p0/test_variant_json_functions.groovy @@ -30,7 +30,7 @@ suite("test_variant_json_functions", "p0") { UNION ALL SELECT 2, '{"a":2,"s":"yz","b":{"c":[3]},"n":{"m":true}}' UNION ALL SELECT 3, '[1,"two",{"three":3}]' UNION ALL SELECT 4, '42' - UNION ALL SELECT 5, '"abc"' + UNION ALL SELECT 5, 'abc' UNION ALL SELECT 6, NULL) t """ From 413eb471410c48cdf55ca417b123ffcf480d9516 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Tue, 20 Oct 2026 12:59:01 +0800 Subject: [PATCH 34/47] [fix](fe) Validate HDFS vault properties before checking connectivity ### What problem does this PR solve? Related PR: #68540 Problem Summary: Creating an HDFS vault with copied S3 properties reached StorageAdapter before generateHdfsParam validated the arguments. The adapter selected an S3 filesystem and reported an unsupported hdfs scheme instead of the expected Invalid argument error, failing the case-sensitive vault regression. Reuse the existing HDFS parameter validation before connectivity checks, including when s3_validity_check is false. ### Release note HDFS vault creation now rejects invalid object-storage properties before attempting filesystem connectivity. ### Check List (For Author) - Test: Added unit coverage for five invalid keys, both letter cases, and enabled/disabled connectivity checks. Targeted Checkstyle and git diff --check passed. FE unit execution was blocked at startup by the local JDK 17 missing lib/modules; cloud regression was not rerun. - Behavior changed: Yes, invalid HDFS vault properties fail before filesystem selection and connectivity. - Does this need documentation: No --- .../doris/catalog/HdfsStorageVault.java | 3 +++ .../cloud/catalog/HdfsStorageVaultTest.java | 24 +++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java index c5851405958e0d..aeaba18d8d3af2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java @@ -103,6 +103,9 @@ public void modifyProperties(ImmutableMap newProperties) throws for (Map.Entry kv : newProperties.entrySet()) { replaceIfEffectiveValue(this.properties, kv.getKey(), kv.getValue()); } + // Reject object-storage properties before they can select a different filesystem + // during the connectivity check, even when that check is disabled. + generateHdfsParam(this.properties); checkConnectivity(this.properties); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java index 2657aede6ad857..dde3696f49b9cf 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java @@ -27,6 +27,7 @@ import org.apache.doris.cloud.rpc.MetaServiceProxy; import org.apache.doris.common.Config; import org.apache.doris.common.DdlException; +import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.system.SystemInfoService; @@ -42,6 +43,7 @@ import org.mockito.Mockito; import java.util.HashSet; +import java.util.Locale; import java.util.Map; import java.util.Set; @@ -262,6 +264,28 @@ public void testCheckConnectivityException() { }); } + @Test + public void testRejectS3PropertiesBeforeConnectivity() { + try (MockedStatic adapter = Mockito.mockStatic(StorageAdapter.class)) { + for (String key : new String[] {"s3.region", "s3.endpoint", "s3.root.path", "provider", "s3.bucket"}) { + for (String invalidKey : new String[] {key, key.toUpperCase(Locale.ROOT)}) { + for (String validityCheck : new String[] {"true", "false"}) { + ImmutableMap properties = ImmutableMap.of( + "FS.DEFAULTFS", "hdfs://localhost:10000", + HdfsStorageVault.PropertyKey.VAULT_PATH_PREFIX, "testInvalidProperties", + HdfsStorageVault.S3_VALIDITY_CHECK, validityCheck, + invalidKey, "invalid"); + HdfsStorageVault vault = new HdfsStorageVault("testHdfsVault", false, false); + IllegalArgumentException exception = Assertions.assertThrows( + IllegalArgumentException.class, () -> vault.modifyProperties(properties)); + Assertions.assertEquals("Invalid argument " + invalidKey, exception.getMessage()); + } + } + } + adapter.verifyNoInteractions(); + } + } + @Test public void testIgnoreCheckConnectivity() throws DdlException { ImmutableMap properties = ImmutableMap.of( From 47872fcd1833f55beea6c140e90241185f5a9143 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Tue, 20 Oct 2026 15:00:52 +0800 Subject: [PATCH 35/47] [fix](storage) Scope provider routing changes to GCS and S3 fallbacks ### What problem does this PR solve? Related PR: #68540 Problem Summary: Treating every registered provider name as a global override changed HDFS and other existing routing paths. A schemeless S3 URI bypassed the HDFS guess and produced a different error, failing the streaming job regression. Provider names also competed inconsistently with explicit filesystem flags. Restore the registry order and filesystem-flag semantics instead of globally prioritizing provider names. Keep explicit S3 interpretation inside S3/MinIO, and retain GCS-owned credential validation and its S3/MinIO fallback declaration. Only declaring plugins are probed when computing fallbacks, avoiding unrelated heuristics before the normal matching loop. Preserve the original regression assertion and add coverage for HDFS precedence, malformed URIs, GCP/GCS aliases, and S3 credential selection at GCS/MinIO endpoints. ### Release note Preserve existing filesystem routing and URI validation while supporting native GCS authentication and explicit S3 authentication at S3-compatible endpoints. ### Check List (For Author) - Test: 50 S3/MinIO/GCS provider tests passed; targeted Checkstyle and git diff --check passed. An isolated harness passed 19 host routing/credential tests using the real manager, adapter and provider code, with host configuration, unused URI helpers and HDFS binding/IO substituted. Full FE UT could not start because the local JDK 17 is incomplete; cloud regression was not rerun. - Behavior changed: Yes, remove global provider-name priority and retain only S3-family selection changes. - Does this need documentation: Yes, filesystem README updated. --- .../doris/fs/FileSystemPluginManager.java | 42 ++++--------- .../doris/fs/FileSystemPluginManagerTest.java | 59 ++++++++++++++++--- .../apache/doris/fs/GcsGuessRoutingTest.java | 41 +++++++++++++ fe/fe-filesystem/README.md | 12 ++-- .../minio/MinioFileSystemProvider.java | 4 ++ .../minio/MinioFileSystemProviderTest.java | 13 ++++ .../filesystem/s3/S3FileSystemProvider.java | 5 ++ .../s3/S3FileSystemProviderTest.java | 12 ++++ 8 files changed, 146 insertions(+), 42 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java index a5564f8ffb1470..4e72cd2427aa6b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java @@ -55,9 +55,8 @@ * 1. ServiceLoader scan (classpath-based built-ins / test overrides) * 2. DirectoryPluginRuntimeManager scan (production plugin directories) * - *

Explicit selections precede heuristic matches, and matching providers may supersede their - * declared fallbacks. Remaining ties retain registration order. Classpath providers have higher - * priority than directory-loaded providers. + *

The first supporting provider is used unless a matching provider supersedes its declared + * heuristic fallbacks. Classpath providers have higher priority than directory-loaded providers. */ public class FileSystemPluginManager { @@ -203,23 +202,15 @@ private boolean hasProviderNamed(String name) { */ public FileSystem createFileSystem(Map properties) throws IOException { validateProperties(properties); - FileSystemProvider named = providerByName(properties.get("provider")); - if (named != null) { - return named.create(properties); - } - for (FileSystemProvider provider : providers) { - if (provider.supportsExplicit(properties)) { - return provider.create(properties); - } - } Set fallbacks = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); for (FileSystemProvider provider : providers) { - if (provider.supports(properties)) { + if (!provider.fallbackProviders().isEmpty() && provider.supports(properties)) { fallbacks.addAll(provider.fallbackProviders()); } } for (FileSystemProvider provider : providers) { - if (!fallbacks.contains(provider.name()) && provider.supports(properties)) { + if ((!fallbacks.contains(provider.name()) || provider.supportsExplicit(properties)) + && provider.supports(properties)) { return provider.create(properties); } } @@ -253,19 +244,12 @@ public void validateProperties(Map properties) { } } - private boolean useGuess(Map properties) { - return !hasAnyExplicitFsSupport(properties) && providerByName(properties.get("provider")) == null; - } - - private static boolean matchesExplicit(FileSystemProvider provider, Map properties) { - return provider.name().equalsIgnoreCase(properties.get("provider")) || provider.supportsExplicit(properties); - } - private Set fallbackProviders(Map properties, Map probeView, boolean useGuess) { Set fallbacks = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); for (FileSystemProvider provider : providers) { - if (matchesExplicit(provider, properties) || (useGuess && provider.supportsGuess(probeView))) { + if (!provider.fallbackProviders().isEmpty() + && (provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView)))) { fallbacks.addAll(provider.fallbackProviders()); } } @@ -274,20 +258,20 @@ private Set fallbackProviders(Map properties, Map provider, Map properties, Map probeView, boolean useGuess, Set fallbacks) { - return matchesExplicit(provider, properties) + return provider.supportsExplicit(properties) || (useGuess && !fallbacks.contains(provider.name()) && provider.supportsGuess(probeView)); } /** * Binds the first matching provider for raw user properties, mirroring * {@code StorageProperties.createPrimary}: fixed priority, explicit {@code fs..support} - * flags first-class, heuristic guesses disabled by an explicit flag or a registered provider name, + * flags first-class, heuristic guesses disabled by an explicit flag, * matched plugins superseding their declared heuristic fallbacks, * and no default fallback — throws when nothing matches. */ public FileSystemProperties bindPrimary(Map properties) { validateProperties(properties); - boolean useGuess = useGuess(properties); + boolean useGuess = !hasAnyExplicitFsSupport(properties); Map probeView = withProbeContext(properties); Set fallbacks = fallbackProviders(properties, probeView, useGuess); for (StorageRegistry.Provider meta : StorageRegistry.Provider.values()) { @@ -325,7 +309,7 @@ public FileSystemProperties bindPrimary(Map properties) { */ public List bindAll(Map properties) { validateProperties(properties); - boolean useGuess = useGuess(properties); + boolean useGuess = !hasAnyExplicitFsSupport(properties); Map probeView = withProbeContext(properties); Set fallbacks = fallbackProviders(properties, probeView, useGuess); List result = new ArrayList<>(); @@ -372,7 +356,7 @@ public List bindAll(Map properties) { result.add(bound); } } - if (!hasAnyExplicitFsSupport(properties) && !hdfsFamilyMatched) { + if (useGuess && !hdfsFamilyMatched) { FileSystemProvider hdfs = providerByName("HDFS"); if (hdfs != null) { result.add(0, hdfs.bind(properties)); @@ -401,7 +385,7 @@ public List bindAll(Map properties) { */ private FileSystemProperties tryBindUnlisted(FileSystemProvider provider, Map properties, boolean useGuess, Map probeView) { - boolean typedMatch = matchesExplicit(provider, properties) + boolean typedMatch = provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView)); if (!typedMatch) { if (useGuess && provider.supports(properties)) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java index df9bb4b7224560..af6a47c1f2b881 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java @@ -144,17 +144,11 @@ public void specificPluginSupersedesFallbackInPrimaryAndAllBindings() throws Exc } @Test - public void explicitSelectionWinsAndMultipleExplicitFlagsRemainSupported() { + public void explicitFlagsWinAndMultipleExplicitFlagsRemainSupported() { FileSystemPluginManager manager = new FileSystemPluginManager(); manager.registerProvider(routingProvider("custom", Set.of("s3"))); manager.registerProvider(routingProvider("S3", Set.of())); Map props = new HashMap<>(Map.of("custom.match", "true", "S3.match", "true")); - props.put("provider", "s3"); - Assertions.assertEquals("S3", manager.bindPrimary(props).providerName()); - Assertions.assertEquals(1, manager.bindAll(props).size()); - Assertions.assertEquals("S3", Assertions.assertThrows(UnsupportedOperationException.class, - () -> manager.createFileSystem(props)).getMessage()); - props.remove("provider"); props.put("fs.s3.support", "true"); Assertions.assertEquals("S3", manager.bindPrimary(props).providerName()); Assertions.assertEquals(1, manager.bindAll(props).size()); @@ -165,17 +159,64 @@ public void explicitSelectionWinsAndMultipleExplicitFlagsRemainSupported() { } @Test - public void namedProviderPreservesDefaultHdfsAndExplicitFlagsDisableIt() { + public void objectStorageGuessPreservesDefaultHdfsAndExplicitFlagsDisableIt() { FileSystemPluginManager manager = new FileSystemPluginManager(); manager.registerProvider(routingProvider("S3", Set.of())); manager.registerProvider(routingProvider("HDFS", Set.of())); - List bindings = manager.bindAll(Map.of("provider", "s3")); + List bindings = manager.bindAll(Map.of("provider", "s3", "S3.match", "true")); Assertions.assertEquals(2, bindings.size()); Assertions.assertEquals("HDFS", bindings.get(0).providerName()); Assertions.assertEquals("S3", bindings.get(1).providerName()); Assertions.assertEquals(1, manager.bindAll(Map.of("fs.s3.support", "true")).size()); } + @Test + public void providerNameDoesNotOverrideExistingMatchOrder() throws Exception { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("S3", Set.of())); + manager.registerProvider(routingProvider("HDFS", Set.of())); + Map props = new HashMap<>(Map.of( + "provider", "S3", "S3.match", "true", "HDFS.match", "true")); + Assertions.assertEquals("HDFS", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(2, manager.bindAll(props).size()); + Assertions.assertEquals("HDFS", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + + props.put("fs.hdfs.support", "true"); + Assertions.assertEquals("HDFS", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("HDFS", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + } + + @Test + public void unrelatedGuessesAreNotEvaluatedBeforePrimaryMatch() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("S3", Set.of())); + manager.registerProvider(new FileSystemProvider() { + @Override + public String name() { + return "unused"; + } + + @Override + public boolean supports(Map properties) { + throw new AssertionError("A later unrelated provider must not be probed"); + } + + @Override + public FileSystem create(Map properties) { + throw new AssertionError("A later unrelated provider must not be created"); + } + + @Override + public boolean supportsGuess(Map properties) { + throw new AssertionError("A later unrelated provider must not be probed"); + } + }); + Assertions.assertEquals("S3", manager.bindPrimary(Map.of("S3.match", "true")).providerName()); + } + @Test public void nonSelectedPluginStillValidatesItsOwnedProperties() { FileSystemPluginManager manager = new FileSystemPluginManager(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java index e44e12a2e01953..d90272469ca9bd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/GcsGuessRoutingTest.java @@ -56,6 +56,47 @@ public void testExplicitS3KeepsAwsAuthenticationInPrimaryAndAllBindings() { } } + @Test + public void testProviderNamesPreserveLegacySchemalessUriError() { + for (String provider : new String[] {"S3", "OSS", "COS", "OBS", "AZURE", "GCS", "GCP"}) { + Map props = properties("https://storage.googleapis.com"); + props.put("provider", provider); + props.put("uri", "bucket/regression/load/data/example_0.csv"); + props.put("s3.access_key", "ak"); + props.put("s3.secret_key", "sk"); + IllegalArgumentException primary = Assertions.assertThrows( + IllegalArgumentException.class, () -> StorageAdapter.of(props), provider); + Assertions.assertTrue(primary.getMessage().contains("Invalid uri: bucket/"), primary.getMessage()); + IllegalArgumentException all = Assertions.assertThrows( + IllegalArgumentException.class, () -> StorageAdapter.ofAll(props), provider); + Assertions.assertTrue(all.getMessage().contains("Invalid uri: bucket/"), all.getMessage()); + } + } + + @Test + public void testGcpAndGcsAliasesSelectTheSameBindings() { + for (String provider : new String[] {"GCP", "GCS", "gcp", "gcs"}) { + Map props = properties("https://storage.googleapis.com"); + props.put("provider", provider); + bindings(props, "GCS"); + props.put("s3.access_key", "ak"); + props.put("s3.secret_key", "sk"); + bindings(props, "GCS"); + } + } + + @Test + public void testExplicitHdfsFlagIsNotOverriddenByProviderName() { + Map props = properties("https://storage.googleapis.com"); + props.put("provider", "S3"); + props.put("fs.hdfs.support", "true"); + props.put("uri", "hdfs://namenode/path"); + Assertions.assertEquals("HDFS", StorageAdapter.of(props).getStorageName()); + List all = StorageAdapter.ofAll(props); + Assertions.assertEquals(1, all.size()); + Assertions.assertEquals("HDFS", all.get(0).getStorageName()); + } + @Test public void testInferredGcsEndpointsBindNativeAndHmacWithoutMinio() { for (String endpoint : ENDPOINTS) { diff --git a/fe/fe-filesystem/README.md b/fe/fe-filesystem/README.md index e189f3c6562205..2c5df636e4f3a6 100644 --- a/fe/fe-filesystem/README.md +++ b/fe/fe-filesystem/README.md @@ -175,17 +175,21 @@ changed. Plugins must be rebuilt against the matching host APIs. ### Overlapping provider matches -The binding registry honors an explicit registered `provider` name or `fs..support` flags before -heuristic matching. A provider can declare `fallbackProviders()` to suppress general-purpose +The binding registry preserves the existing provider order and `fs..support` flag semantics. +It does not treat a `provider` property as a global override; each plugin owns its interpretation +of that property. A provider can declare `fallbackProviders()` to suppress general-purpose heuristic matches when it also matches. This applies to both primary and multi-provider binding; -explicit matches are never suppressed. Providers without this declaration retain their existing -multi-match behavior. The default HDFS binding for catalogs is preserved. +explicit filesystem flags are never suppressed. Providers without this declaration retain their +existing multi-match behavior. The default HDFS binding and HDFS URI validation are preserved. `validateProperties()` runs on every loaded provider before selection, so invalid provider-owned credentials cannot silently disappear when another plugin is chosen. It must be deterministic and perform no I/O. Forced provider binding validates with the effective provider name too. GCS owns its endpoint/credential recognition and declares S3 and MinIO as heuristic fallbacks. +Within this family, `provider=S3` retains S3 authentication and disables GCS/MinIO guessing; +`provider=GCS` and `provider=GCP` are both recognized by GCS. These choices do not override the +existing HDFS, OSS, COS, OBS, or Azure routing order. S3 and its shared protocol library do not depend on GCS authentication classes. Filesystem plugin API 5.0 includes these two default hooks; all plugins must be rebuilt for that host API. diff --git a/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java index f08e741dd39b2f..92e16d6b4d5c5d 100644 --- a/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java @@ -89,6 +89,10 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { + // Match the raw supports() path: an explicit S3 request must not also bind MinIO. + if (S3CompatSignals.hasExplicitS3Request(properties)) { + return false; + } // Verbatim port of fe-core MinioProperties.guessIsMe: MinIO is the "any other // S3-compatible" fallback — it claims the map iff none of Azure/COS/OSS/S3 would claim // it AND at least one identifier key is present. The sibling guesses are replicated diff --git a/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java index 1058dedc7fab0b..33c7405936eacf 100644 --- a/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java @@ -108,6 +108,19 @@ void supports_rejectsForeignDialects() { Assertions.assertFalse(provider.supports(plainS3)); } + @Test + void supportsGuess_doesNotAlsoBindExplicitS3() { + for (String endpoint : new String[] {"https://storage.us-east1.rep.googleapis.com", + "http://minio.local:9000"}) { + Map props = new HashMap<>(); + props.put("s3.endpoint", endpoint); + Assertions.assertTrue(provider.supportsGuess(props), endpoint); + props.put("provider", "S3"); + Assertions.assertFalse(provider.supportsGuess(props), endpoint); + Assertions.assertFalse(provider.supports(props), endpoint); + } + } + @Test void supportsGuess_yieldsToAzureSovereignCloudHostSuffixes() { // The Azure claim is host-suffix based (all sovereign clouds, blob AND dfs), not a diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java index a4f95a85fd0a66..38e434945394ef 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProvider.java @@ -115,6 +115,11 @@ public boolean supportsExplicit(Map properties) { @Override public boolean supportsGuess(Map properties) { + // Keep an explicit S3 request on the S3 credential path even at a GCS/MinIO endpoint. + // This is local to the S3 family; other providers retain their existing match order. + if (S3CompatSignals.hasExplicitS3Request(properties)) { + return true; + } // Port of fe-core S3Properties.guessIsMe on raw props. String endpoint = null; for (String name : GUESS_ENDPOINT_NAMES) { diff --git a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java index c81f119386bc97..1d4e771a777009 100644 --- a/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-s3/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemProviderTest.java @@ -127,6 +127,18 @@ void supports_yieldsToExplicitlyRequestedDedicatedDialects() { } } + @Test + void supportsGuess_keepsExplicitS3AtGcsAndMinioEndpoints() { + for (String endpoint : new String[] {"https://storage.googleapis.com", + "https://storage.us-east1.rep.googleapis.com", "http://minio.local:9000"}) { + for (String name : new String[] {"S3", "s3"}) { + Map props = Map.of("provider", name, "s3.endpoint", endpoint); + Assertions.assertTrue(provider.supportsGuess(props), endpoint); + Assertions.assertTrue(provider.supports(props), endpoint); + } + } + } + @Test void supports_keepsGoogleapisMapWhenUserExplicitlyRequestsS3() { // An explicit fs.s3.support=true / provider=S3 overrides the dialect guess: the user has From 4526bcd5312e336dcffc16f7bb2515e20248d767 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Wed, 21 Oct 2026 11:33:20 +0800 Subject: [PATCH 36/47] [fix](fe) Preserve HDFS vault Kerberos validation errors ### What problem does this PR solve? Related PR: #68540 Problem Summary: Calling generateHdfsParam before connectivity validation changed the error for Kerberos vaults without credentials from the filesystem's established missing-principal-or-keytab message to a metadata serialization error. This failed test_create_vault_with_kerberos in VaultP0 build 1074733. Extract only the existing forbidden S3-property checks for early validation and reuse them during metadata generation. Leave credential validation in the filesystem binding path. Add tests for each missing-credential combination. ### Release note Preserve the existing error message when creating an HDFS vault with incomplete Kerberos credentials, while continuing to reject misplaced S3 properties before connectivity checks. ### Check List (For Author) - Test: FE Checkstyle and git diff --check passed. Isolated validation compiled the unchanged production vault and HDFS provider/authentication sources and ran the new test method with minimal peripheral fixtures: three missing-credential combinations, 20 forbidden-key/case/connectivity combinations, and complete Kerberos metadata preservation passed. The new test fails on the original code with the same error as TeamCity. Standard run-fe-ut.sh was attempted but blocked by the local broken JDK 17; supplementary HDFS module tests were blocked by an uncached Jetty dependency. Cloud regression was not rerun. - Behavior changed: Yes, restore the established Kerberos validation order and error contract; early S3-property rejection is retained. - Does this need documentation: No --- .../doris/catalog/HdfsStorageVault.java | 37 ++++++++++--------- .../cloud/catalog/HdfsStorageVaultTest.java | 21 +++++++++++ 2 files changed, 40 insertions(+), 18 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java index aeaba18d8d3af2..62167919196afc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java @@ -38,6 +38,7 @@ import java.io.IOException; import java.util.Arrays; import java.util.HashSet; +import java.util.Locale; import java.util.Map; import java.util.Set; import java.util.stream.Collectors; @@ -105,7 +106,8 @@ public void modifyProperties(ImmutableMap newProperties) throws } // Reject object-storage properties before they can select a different filesystem // during the connectivity check, even when that check is disabled. - generateHdfsParam(this.properties); + // Leave Kerberos validation in filesystem binding to preserve its error message. + this.properties.keySet().forEach(HdfsStorageVault::checkForbiddenProperty); checkConnectivity(this.properties); } @@ -195,23 +197,7 @@ public static Cloud.HdfsVaultInfo generateHdfsParam(Map properti } else if (property.getKey().equalsIgnoreCase(StorageVault.PropertyKey.VAULT_NAME)) { continue; } else { - // Get rid of copy and paste from create s3 vault stmt - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.REGION), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.ENDPOINT), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.ROOT_PATH), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.PROVIDER), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.BUCKET), - "Invalid argument %s", property.getKey()); - + checkForbiddenProperty(property.getKey()); if (!NON_HDFS_CONF_PROPERTY_KEYS.contains(property.getKey().toLowerCase())) { Cloud.HdfsBuildConf.HdfsConfKVPair.Builder conf = Cloud.HdfsBuildConf.HdfsConfKVPair.newBuilder(); conf.setKey(property.getKey()); @@ -222,4 +208,19 @@ public static Cloud.HdfsVaultInfo generateHdfsParam(Map properti } return hdfsVaultInfoBuilder.setBuildConf(hdfsConfBuilder.build()).build(); } + + private static void checkForbiddenProperty(String key) { + // Get rid of copy and paste from create s3 vault stmt. + String lowerCaseKey = key.toLowerCase(Locale.ROOT); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.REGION), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.ENDPOINT), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.ROOT_PATH), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.PROVIDER), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.BUCKET), + "Invalid argument %s", key); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java index dde3696f49b9cf..03d292b7442d4d 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java @@ -43,6 +43,7 @@ import org.mockito.Mockito; import java.util.HashSet; +import java.util.List; import java.util.Locale; import java.util.Map; import java.util.Set; @@ -286,6 +287,26 @@ public void testRejectS3PropertiesBeforeConnectivity() { } } + @Test + public void testKerberosValidationBeforeConnectivity() { + for (Map credentials : List.>of( + ImmutableMap.of(), + ImmutableMap.of(HdfsStorageVault.PropertyKey.HADOOP_KERBEROS_PRINCIPAL, "hadoop/localhost@EXAMPLE.COM"), + ImmutableMap.of(HdfsStorageVault.PropertyKey.HADOOP_KERBEROS_KEYTAB, "/path/to/test.keytab"))) { + ImmutableMap properties = ImmutableMap.builder() + .put(HdfsStorageVault.PropertyKey.HADOOP_FS_NAME, "hdfs://localhost:10000") + .put(HdfsStorageVault.PropertyKey.VAULT_PATH_PREFIX, "testKerberosProperties") + .put(HdfsStorageVault.PropertyKey.HADOOP_SECURITY_AUTHENTICATION, "kerberos") + .putAll(credentials) + .build(); + HdfsStorageVault vault = new HdfsStorageVault("testHdfsVault", false, false); + IllegalArgumentException exception = Assertions.assertThrows( + IllegalArgumentException.class, () -> vault.modifyProperties(properties)); + Assertions.assertEquals("HDFS authentication type is kerberos, but principal or keytab is not set.", + exception.getMessage()); + } + } + @Test public void testIgnoreCheckConnectivity() throws DdlException { ImmutableMap properties = ImmutableMap.of( From 4f00011b8fd31552c48918c1c2597b1c3fc5c119 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Wed, 21 Oct 2026 16:09:03 +0800 Subject: [PATCH 37/47] [fix](storage) Exchange service-account ADC for GCS OAuth tokens ### What problem does this PR solve? Related PR: #68540 Problem Summary: The generic google-cloud-cpp adapter returns scoped self-signed JWTs for service-account ADC files, which GCS rejects. Resolve the SDK ADC file paths and construct service-account credentials with JWTs disabled and the requested OAuth scope. Keep SDK token caching, CA configuration and signing identity, and preserve generic ADC and impersonation handling for other sources. Add tests using generated RSA keys, actual ADC files and a local OAuth endpoint for token exchange, caching, scopes, exchange failure and invalid credentials. ### Release note GCS service-account ADC now exchanges OAuth access tokens for object access. ### Check List (For Author) - Test: clang-format 16, build hygiene and git diff --check passed. BE unit tests could not start: the configured JDK 17 lacks lib/modules, and the available JDK 20 is rejected by the test runner. Google Cloud C++ libraries are also absent from the installed thirdparty bundle. No live GCP test ran. - Behavior changed: Yes, direct service-account ADC uses OAuth instead of JWTs. - Does this need documentation: No --- .../auth/gcp/gcp_token_provider_test.cpp | 218 ++++++++++++++++++ .../auth/gcp/gcp_token_provider.cpp | 63 ++++- 2 files changed, 275 insertions(+), 6 deletions(-) create mode 100644 be/test/common/auth/gcp/gcp_token_provider_test.cpp diff --git a/be/test/common/auth/gcp/gcp_token_provider_test.cpp b/be/test/common/auth/gcp/gcp_token_provider_test.cpp new file mode 100644 index 00000000000000..c1cc7d9fc10d68 --- /dev/null +++ b/be/test/common/auth/gcp/gcp_token_provider_test.cpp @@ -0,0 +1,218 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "cpp/obj-client/auth/gcp/gcp_token_provider.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "service/http/ev_http_server.h" +#include "service/http/http_channel.h" +#include "service/http/http_handler.h" +#include "service/http/http_headers.h" +#include "service/http/http_request.h" +#include "service/http/http_status.h" + +namespace doris { +namespace { + +class OAuthTokenHandler : public HttpHandler { +public: + void handle(HttpRequest* request) override { + std::lock_guard lock(mutex); + requests.push_back(request->get_request_body()); + request->add_output_header(HttpHeaders::CONTENT_TYPE, "application/json"); + if (reject) { + HttpChannel::send_reply(request, HttpStatus::UNAUTHORIZED, + R"({"error":"invalid_grant"})"); + return; + } + HttpChannel::send_reply( + request, + R"({"access_token":"oauth-access-token","expires_in":3600,"token_type":"Bearer"})"); + } + + std::mutex mutex; + std::vector requests; + bool reject = false; +}; + +class GcpTokenProviderTest : public testing::Test { +protected: + void SetUp() override { + for (size_t i = 0; i < ENV_NAMES.size(); ++i) { + const char* value = std::getenv(ENV_NAMES[i]); + saved_env[i] = value == nullptr ? std::nullopt : std::optional(value); + unsetenv(ENV_NAMES[i]); + } + char path[] = "/tmp/doris_gcp_adc_XXXXXX"; + int fd = mkstemp(path); + ASSERT_GE(fd, 0); + close(fd); + adc_path = path; + ASSERT_EQ(setenv("GOOGLE_APPLICATION_CREDENTIALS", adc_path.c_str(), 1), 0); + ASSERT_TRUE(server.register_handler(POST, "/token", &handler)); + server.start(); + ASSERT_GT(server.get_real_port(), 0); + token_uri = "http://127.0.0.1:" + std::to_string(server.get_real_port()) + "/token"; + + // This is a freshly generated test key, never a real Google identity or secret. + std::unique_ptr context( + EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, nullptr), EVP_PKEY_CTX_free); + ASSERT_NE(context, nullptr); + ASSERT_EQ(EVP_PKEY_keygen_init(context.get()), 1); + ASSERT_EQ(EVP_PKEY_CTX_set_rsa_keygen_bits(context.get(), 2048), 1); + EVP_PKEY* raw_key = nullptr; + ASSERT_EQ(EVP_PKEY_keygen(context.get(), &raw_key), 1); + std::unique_ptr key(raw_key, EVP_PKEY_free); + std::unique_ptr bio(BIO_new(BIO_s_mem()), BIO_free); + ASSERT_NE(bio, nullptr); + ASSERT_EQ(PEM_write_bio_PrivateKey(bio.get(), key.get(), nullptr, nullptr, 0, nullptr, + nullptr), + 1); + char* pem = nullptr; + auto length = BIO_get_mem_data(bio.get(), &pem); + private_key.assign(pem, length); + write_service_account(); + } + + void TearDown() override { + std::remove(adc_path.c_str()); + for (size_t i = 0; i < ENV_NAMES.size(); ++i) { + if (saved_env[i]) { + setenv(ENV_NAMES[i], saved_env[i]->c_str(), 1); + } else { + unsetenv(ENV_NAMES[i]); + } + } + } + + void write_service_account() { + rapidjson::Document document; + document.SetObject(); + auto& allocator = document.GetAllocator(); + document.AddMember("type", "service_account", allocator); + document.AddMember("client_email", "test@example.iam.gserviceaccount.com", allocator); + document.AddMember("private_key_id", "test-key-id", allocator); + document.AddMember("private_key", rapidjson::Value(private_key.c_str(), allocator), + allocator); + document.AddMember("token_uri", rapidjson::Value(token_uri.c_str(), allocator), allocator); + rapidjson::StringBuffer buffer; + rapidjson::Writer writer(buffer); + document.Accept(writer); + std::ofstream output(adc_path); + output << buffer.GetString(); + ASSERT_TRUE(output.good()); + } + + void expect_exchange_scope(const std::string& expected_scope) { + std::lock_guard lock(handler.mutex); + ASSERT_EQ(handler.requests.size(), 1); + const auto& body = handler.requests.front(); + EXPECT_NE(body.find("grant_type="), std::string::npos); + auto assertion = body.find("assertion="); + ASSERT_NE(assertion, std::string::npos); + auto start = body.find('.', assertion); + ASSERT_NE(start, std::string::npos); + auto end = body.find('.', start + 1); + ASSERT_NE(end, std::string::npos); + auto encoded = body.substr(start + 1, end - start - 1); + std::replace(encoded.begin(), encoded.end(), '-', '+'); + std::replace(encoded.begin(), encoded.end(), '_', '/'); + while (encoded.size() % 4 != 0) { + encoded.push_back('='); + } + std::string decoded(encoded.size(), '\0'); + int length = EVP_DecodeBlock(reinterpret_cast(decoded.data()), + reinterpret_cast(encoded.data()), + encoded.size()); + ASSERT_GT(length, 0); + rapidjson::Document payload; + payload.Parse(decoded.c_str()); + ASSERT_FALSE(payload.HasParseError()); + ASSERT_TRUE(payload.HasMember("scope")); + EXPECT_EQ(std::string(payload["scope"].GetString()), expected_scope); + ASSERT_TRUE(payload.HasMember("aud")); + EXPECT_EQ(std::string(payload["aud"].GetString()), token_uri); + } + + static constexpr std::array ENV_NAMES { + "GOOGLE_APPLICATION_CREDENTIALS", "GOOGLE_GCLOUD_ADC_PATH_OVERRIDE", + "GOOGLE_CLOUD_CPP_EXPERIMENTAL_DISABLE_SELF_SIGNED_JWT"}; + std::array, ENV_NAMES.size()> saved_env; + std::string adc_path; + std::string private_key; + std::string token_uri; + OAuthTokenHandler handler; + EvHttpServer server {"127.0.0.1", 0}; +}; + +TEST_F(GcpTokenProviderTest, ServiceAccountAdcExchangesAndCachesOAuthToken) { + GcpTokenProvider provider(GcpCredentialConfig {}); + EXPECT_EQ(provider.get_token(), "oauth-access-token"); + EXPECT_EQ(provider.get_token(), "oauth-access-token"); + EXPECT_EQ(provider.get_service_account_email(), "test@example.iam.gserviceaccount.com"); + expect_exchange_scope("https://www.googleapis.com/auth/devstorage.read_write"); + EXPECT_EQ(std::getenv("GOOGLE_CLOUD_CPP_EXPERIMENTAL_DISABLE_SELF_SIGNED_JWT"), nullptr); +} + +TEST_F(GcpTokenProviderTest, WellKnownAdcExchangesOAuthTokenWithCloudPlatformScope) { + unsetenv("GOOGLE_APPLICATION_CREDENTIALS"); + ASSERT_EQ(setenv("GOOGLE_GCLOUD_ADC_PATH_OVERRIDE", adc_path.c_str(), 1), 0); + GcpTokenProvider provider(GcpCredentialConfig {}, "", GcpTokenScope::CloudPlatform); + EXPECT_EQ(provider.get_token(), "oauth-access-token"); + expect_exchange_scope("https://www.googleapis.com/auth/cloud-platform"); +} + +TEST_F(GcpTokenProviderTest, OAuthFailureDoesNotReturnSelfSignedJwt) { + { + std::lock_guard lock(handler.mutex); + handler.reject = true; + } + GcpTokenProvider provider(GcpCredentialConfig {}); + EXPECT_FALSE(provider.get_token().has_value()); + expect_exchange_scope("https://www.googleapis.com/auth/devstorage.read_write"); +} + +TEST_F(GcpTokenProviderTest, InvalidServiceAccountAdcDoesNotFallBackToAnotherIdentity) { + std::ofstream(adc_path) + << R"({"type":"service_account","client_email":"test@example.iam.gserviceaccount.com"})"; + GcpTokenProvider provider(GcpCredentialConfig {}); + EXPECT_FALSE(provider.get_token().has_value()); + std::lock_guard lock(handler.mutex); + EXPECT_TRUE(handler.requests.empty()); +} + +} // namespace +} // namespace doris diff --git a/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp b/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp index 9716f786a3a384..42174b05e8ec98 100644 --- a/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp +++ b/common/cpp/obj-client/auth/gcp/gcp_token_provider.cpp @@ -19,8 +19,18 @@ #include #include +#include +#include +#include +#include +#include #include +#include +#include +#include +#include +#include #include namespace doris { @@ -29,9 +39,46 @@ namespace { constexpr char GCS_READ_WRITE_SCOPE[] = "https://www.googleapis.com/auth/devstorage.read_write"; constexpr char CLOUD_PLATFORM_SCOPE[] = "https://www.googleapis.com/auth/cloud-platform"; -std::shared_ptr make_credentials(const GcpCredentialConfig& credential, - const std::string& ca_cert_path, - GcpTokenScope token_scope) { +std::shared_ptr make_default_storage_credentials( + const google::cloud::Options& options) { + namespace oauth2 = google::cloud::oauth2_internal; + auto path = oauth2::GoogleAdcFilePathFromEnvVarOrEmpty(); + if (path.empty()) { + path = oauth2::GoogleAdcFilePathFromWellKnownPathOrEmpty(); + } + std::ifstream input(path); + std::string contents(std::istreambuf_iterator {input}, {}); + rapidjson::Document document; + document.Parse(contents.data(), contents.size()); + // The generic SDK adapter emits scoped self-signed JWTs for service-account + // JSON keys, which GCS rejects. Keep other ADC types (and their errors) on + // the SDK's normal resolution path, including federation and VM identities. + if (document.HasParseError() || !document.IsObject() || !document.HasMember("type") || + !document["type"].IsString() || + std::string_view(document["type"].GetString(), document["type"].GetStringLength()) != + "service_account") { + return google::cloud::rest_internal::MapCredentials( + *google::cloud::MakeGoogleDefaultCredentials(options)); + } + auto info = oauth2::ParseServiceAccountCredentials(contents, path); + if (!info) { + return std::make_shared(info.status()); + } + info->enable_self_signed_jwt = false; + const auto& scopes = options.get(); + info->scopes = std::set(scopes.begin(), scopes.end()); + return oauth2::Decorate(std::make_shared( + *info, options, + [](const google::cloud::Options& opts) { + return google::cloud::rest_internal::MakeDefaultRestClient( + "", opts); + }), + options); +} + +std::shared_ptr make_credentials( + const GcpCredentialConfig& credential, const std::string& ca_cert_path, + GcpTokenScope token_scope) { google::cloud::Options options; if (!ca_cert_path.empty()) { options.set(ca_cert_path); @@ -51,6 +98,11 @@ std::shared_ptr make_credentials(const GcpCredential std::shared_ptr credentials; switch (credential.provider_type) { case GcpCredentialProviderType::Default: + // A direct service-account ADC token goes to GCS. Impersonation sources + // instead authenticate to IAM, which accepts the SDK's self-signed JWTs. + if (credential.impersonation_service_account.empty()) { + return make_default_storage_credentials(options); + } credentials = google::cloud::MakeGoogleDefaultCredentials(options); break; case GcpCredentialProviderType::ComputeEngine: @@ -64,15 +116,14 @@ std::shared_ptr make_credentials(const GcpCredential std::move(credentials), credential.impersonation_service_account, std::move(impersonation_options)); } - return credentials; + return google::cloud::rest_internal::MapCredentials(*credentials); } } // namespace GcpTokenProvider::GcpTokenProvider(const GcpCredentialConfig& credential, const std::string& ca_cert_path, GcpTokenScope token_scope) - : GcpTokenProvider(google::cloud::rest_internal::MapCredentials( - *make_credentials(credential, ca_cert_path, token_scope))) {} + : GcpTokenProvider(make_credentials(credential, ca_cert_path, token_scope)) {} GcpTokenProvider::GcpTokenProvider( std::shared_ptr credentials) From 21766f515e82a843be7ef680dde8923694fdc00b Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Wed, 21 Oct 2026 16:09:26 +0800 Subject: [PATCH 38/47] [fix](fe) Map MinIO resource providers for storage policy pushes ### What problem does this PR solve? Related PR: #68540 Problem Summary: An accepted MINIO resource provider is absent from the Thrift object storage enum, so converting it throws and prevents the entire storage policy/resource update batch from being sent. Map MINIO and OZONE filesystem dialects to the existing AWS-compatible backend while preserving other wire providers. Cover aliases plus ALTER, resource serialization/replay and a batch containing an unreferenced MinIO resource, another resource and a policy drop. ### Release note MinIO resource configuration no longer blocks batched storage policy updates. ### Check List (For Author) - Test: FE Core Checkstyle (zero violations) and git diff --check passed. Targeted FE unit tests could not start because the configured JDK 17 lacks lib/modules and the available JDK 20 is rejected. Offline FE reactor dependencies are incomplete. No integration tests ran. - Behavior changed: Yes, MinIO and Ozone policy providers use the AWS wire type. - Does this need documentation: No --- .../datasource/storage/S3ThriftAdapter.java | 16 +++++++- .../apache/doris/catalog/S3ResourceTest.java | 38 +++++++++++++++++++ .../storage/S3ThriftAdapterParityTest.java | 25 ++++++++++++ 3 files changed, 77 insertions(+), 2 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java index bbbb3478946473..1d02992b145a48 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ThriftAdapter.java @@ -74,8 +74,20 @@ public static TS3StorageParam getS3TStorageParam(Map properties) TS3StorageParam s3Info = new TS3StorageParam(); if (StringUtils.isNotBlank(properties.get("provider"))) { String provider = properties.get("provider").trim().toUpperCase(Locale.ROOT); - s3Info.setProvider("S3".equals(provider) ? TObjStorageType.AWS - : "GCS".equals(provider) ? TObjStorageType.GCP : TObjStorageType.valueOf(provider)); + // Filesystem dialect names such as MINIO have no Thrift enum value. + // They use the same AWS-compatible backend as S3 and must not abort a policy batch. + switch (provider) { + case "S3": + case "MINIO": + case "OZONE": + s3Info.setProvider(TObjStorageType.AWS); + break; + case "GCS": + s3Info.setProvider(TObjStorageType.GCP); + break; + default: + s3Info.setProvider(TObjStorageType.valueOf(provider)); + } } if (properties.containsKey(ROLE_ARN)) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 1456b263f115c6..20b378d47c7000 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -34,6 +34,8 @@ import org.apache.doris.qe.ConnectContext; import org.apache.doris.task.PushStoragePolicyTask; import org.apache.doris.thrift.TGcpCredential; +import org.apache.doris.thrift.TObjStorageType; +import org.apache.doris.thrift.TPushStoragePolicyReq; import org.apache.doris.thrift.TS3StorageParam; import com.google.common.base.Strings; @@ -705,6 +707,42 @@ public void testLegacyGcpDefaultChainSurvivesReplayAndPolicyPush() throws Except } } + @Test + public void testMinioAlterSurvivesReplayAndBatchedPolicyPush() throws Exception { + S3Resource minio = new S3Resource("minio_resource"); + minio.id = 1L; + minio.version = 0L; + minio.setProperties(ImmutableMap.copyOf(s3Properties)); + minio.modifyProperties(ImmutableMap.of("provider", "MINIO", "s3_validity_check", "false")); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (DataOutputStream output = new DataOutputStream(bytes)) { + minio.write(output); + } + S3Resource restored; + try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + restored = (S3Resource) Resource.read(input); + } + Assertions.assertEquals("MINIO", restored.getProperty("provider")); + S3Resource unrelated = new S3Resource("unrelated_resource"); + unrelated.id = 2L; + unrelated.version = 0L; + unrelated.setProperties(ImmutableMap.copyOf(s3Properties)); + unrelated.modifyProperties(ImmutableMap.of(S3ResourceCompat.CONNECTION_TIMEOUT_MS, "2000")); + // The MinIO resource is unreferenced but participates in the same update as other resources/drops. + PushStoragePolicyTask task = new PushStoragePolicyTask(1L, List.of(), + List.of(restored, unrelated), List.of(9L)); + TPushStoragePolicyReq request = task.toThrift(); + Assertions.assertEquals(2, request.getResourceSize()); + Assertions.assertEquals(1L, request.getResource().get(0).getId()); + Assertions.assertEquals(2L, request.getResource().get(1).getId()); + Assertions.assertEquals(List.of(9L), request.getDroppedStoragePolicy()); + TS3StorageParam minioThrift = request.getResource().get(0).getS3StorageParam(); + Assertions.assertEquals(TObjStorageType.AWS, minioThrift.getProvider()); + Assertions.assertEquals(s3AccessKey, minioThrift.getAk()); + Assertions.assertEquals(s3SecretKey, minioThrift.getSk()); + Assertions.assertEquals(2000, request.getResource().get(1).getS3StorageParam().getConnTimeoutMs()); + } + @Test public void testPolicyPushReleasesResourceLockOnInvalidAuthentication() throws Exception { S3Resource resource = new S3Resource("invalid_gcp_resource"); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/storage/S3ThriftAdapterParityTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/storage/S3ThriftAdapterParityTest.java index 5bc69f8b37c13b..f13c6fdf0d63d2 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/storage/S3ThriftAdapterParityTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/storage/S3ThriftAdapterParityTest.java @@ -18,6 +18,7 @@ package org.apache.doris.datasource.storage; import org.apache.doris.thrift.TCredProviderType; +import org.apache.doris.thrift.TObjStorageType; import org.apache.doris.thrift.TS3StorageParam; import org.junit.jupiter.api.Assertions; @@ -133,6 +134,30 @@ public void testAssumeRoleWithLegacyEnvProviderTypeKey() { "AWS_CREDENTIALS_PROVIDER_TYPE", "SYSTEM_PROPERTIES"))); } + @Test + public void testFilesystemProviderAliasesUseBackendTypes() { + for (String provider : new String[] {"S3", "MINIO", "minio", " OZONE ", "AWS"}) { + TS3StorageParam thrift = S3ThriftAdapter.getS3TStorageParam(map( + "provider", provider, + "s3.endpoint", "http://minio.example.com:9000", + "s3.access_key", "myAk", + "s3.secret_key", "mySk")); + Assertions.assertEquals(TObjStorageType.AWS, thrift.getProvider()); + Assertions.assertEquals("myAk", thrift.getAk()); + Assertions.assertEquals("mySk", thrift.getSk()); + } + for (String provider : new String[] {"GCP", "GCS"}) { + Assertions.assertEquals(TObjStorageType.GCP, + S3ThriftAdapter.getS3TStorageParam(map("provider", provider)).getProvider()); + } + for (TObjStorageType provider : new TObjStorageType[] { + TObjStorageType.OSS, TObjStorageType.COS, TObjStorageType.OBS, + TObjStorageType.BOS, TObjStorageType.TOS, TObjStorageType.AZURE}) { + Assertions.assertEquals(provider, + S3ThriftAdapter.getS3TStorageParam(map("provider", provider.name())).getProvider()); + } + } + @Test public void testEmptyMap() { Assertions.assertEquals(expected(null, null), From b4068153fb943e6132b92960ee85f46d32d46229 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Wed, 21 Oct 2026 16:59:56 +0800 Subject: [PATCH 39/47] [fix](fe) Normalize resource aliases for the GCS provider spelling ### What problem does this PR solve? Related PR: apache/doris#68540 Problem Summary: ALTER RESOURCE accepts GCS as a GCP provider alias, but resource normalization discarded gs.* updates for that spelling. Rotating HMAC keys while selecting GCS could succeed while retaining the old keys. Normalize aliases for both spellings, including case variants. Extend provider and resource coverage and verify key rotation through metadata serialization/readback, filesystem binding, and storage policy pushes. ### Release note GCS resource ALTER correctly applies gs.* properties when provider=GCS. ### Check List (For Author) - Test: FE Core and GCS module Checkstyle and git diff --check passed. Added and extended unit coverage. The standard run-fe-ut.sh invocation could not start because the configured JDK 17 is missing lib/modules; unit tests and live GCP integration tests were not run. - Behavior changed: Yes, GCS and GCP apply resource aliases consistently. - Does this need documentation: No --- .../apache/doris/catalog/S3ResourceTest.java | 40 ++++++++++++++++++- .../filesystem/gcs/GcsFileSystemProvider.java | 2 +- .../gcs/GcsFileSystemProviderTest.java | 12 +++--- 3 files changed, 47 insertions(+), 7 deletions(-) diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 20b378d47c7000..2461ebecbba446 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -763,7 +763,7 @@ public void testPolicyPushReleasesResourceLockOnInvalidAuthentication() throws E @Test public void testGsAliasesMatchConnectorAndResourceProtocols() throws Exception { - for (String provider : new String[] {"GCP", "gcp"}) { + for (String provider : new String[] {"GCP", "gcp", "GCS", "gcs"}) { Map properties = new HashMap<>(s3Properties); properties.put("provider", provider); properties.put("gs.endpoint", "https://storage.googleapis.com"); @@ -813,6 +813,44 @@ public void testGsAliasesMatchConnectorAndResourceProtocols() throws Exception { } } + @Test + public void testGcsProviderAlterRotatesKeysAcrossReplayAndPolicyPush() throws Exception { + for (String provider : new String[] {"GCS", "gcs"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + S3Resource resource = new S3Resource("gcs_resource"); + resource.setProperties(ImmutableMap.copyOf(properties)); + resource.references.put("policy", Resource.ReferenceType.POLICY); + resource.modifyProperties(ImmutableMap.of("provider", provider, + "gs.access_key", "rotated-access", "gs.secret_key", "rotated-secret", + "s3_validity_check", "false")); + Assertions.assertEquals("rotated-access", resource.getProperty(S3ResourceCompat.ACCESS_KEY)); + Assertions.assertEquals("rotated-secret", resource.getProperty(S3ResourceCompat.SECRET_KEY)); + + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (DataOutputStream output = new DataOutputStream(bytes)) { + resource.write(output); + } + S3Resource restored; + try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + restored = (S3Resource) Resource.read(input); + } + Assertions.assertEquals(resource.getCopiedProperties(), restored.getCopiedProperties()); + Assertions.assertEquals(provider, restored.getProperty("provider")); + Assertions.assertFalse(restored.getCopiedProperties().containsKey("gs.access_key")); + Assertions.assertFalse(restored.getCopiedProperties().containsKey("gs.secret_key")); + S3CompatibleFileSystemProperties connector = (S3CompatibleFileSystemProperties) + StorageAdapter.of(restored.getCopiedProperties()).getSpiProperties(); + Assertions.assertEquals("rotated-access", connector.getAccessKey()); + Assertions.assertEquals("rotated-secret", connector.getSecretKey()); + PushStoragePolicyTask task = new PushStoragePolicyTask(1L, List.of(), List.of(restored), List.of()); + TS3StorageParam thrift = task.toThrift().getResource().get(0).getS3StorageParam(); + Assertions.assertEquals(TObjStorageType.GCP, thrift.getProvider()); + Assertions.assertEquals("rotated-access", thrift.getAk()); + Assertions.assertEquals("rotated-secret", thrift.getSk()); + } + } + @Test public void testBlankGsCredentialAndConnectionAliases() throws Exception { for (String blank : new String[] {"", " \t"}) { diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java index 0b5920e888e7d7..31dc2ee37dcb0f 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java @@ -74,7 +74,7 @@ public Map normalizeProperties(Map properties, M selected = "GCP"; normalized.put("provider", selected); } - if ("GCP".equalsIgnoreCase(selected)) { + if ("GCP".equalsIgnoreCase(selected) || STORAGE_TYPE_GCS.equalsIgnoreCase(selected)) { RESOURCE_ALIASES.forEach((alias, key) -> { String value = normalized.remove(alias); if (value != null && !value.isBlank()) { diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java index bc9a749c33cd5d..809402292afebe 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java @@ -231,13 +231,15 @@ void supportsGuess_respectsExplicitS3AndRecognizesRegionalEndpoints() { @Test void normalizePropertiesUsesEffectiveProviderForPartialAlter() { - Map stored = Map.of("provider", "GCP", "s3.endpoint", "old.endpoint"); Map patch = Map.of("gs.endpoint", "storage.googleapis.com", "gs.connection.maximum", "25", "gs.impersonation_service_account", ""); - Map normalized = provider.normalizeProperties(patch, stored); - Assertions.assertEquals(Map.of("s3.endpoint", "storage.googleapis.com", "s3.connection.maximum", "25", - "gs.impersonation_service_account", ""), normalized); - Assertions.assertEquals("old.endpoint", stored.get("s3.endpoint")); + for (String selected : new String[] {"GCP", "gcp", "GCS", "gcs"}) { + Map stored = Map.of("provider", selected, "s3.endpoint", "old.endpoint"); + Map normalized = provider.normalizeProperties(patch, stored); + Assertions.assertEquals(Map.of("s3.endpoint", "storage.googleapis.com", "s3.connection.maximum", "25", + "gs.impersonation_service_account", ""), normalized); + Assertions.assertEquals("old.endpoint", stored.get("s3.endpoint")); + } Assertions.assertTrue(patch.containsKey("gs.endpoint")); Assertions.assertTrue(provider.clearablePropertyKeys().contains("gs.impersonation_service_account")); Assertions.assertTrue(provider.modifiableCredentialPropertyKeys().contains("gs.credential_provider_type")); From cd814029aad66e1f753e12b612591ed09297ef64 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Thu, 22 Oct 2026 11:07:52 +0800 Subject: [PATCH 40/47] [fix](storage) Fix GCS resource validation and OAuth test setup ### What problem does this PR solve? Related PR: #68540 Problem Summary: S3ResourceTest rejects provider=GCS after filesystem alias normalization because the resource validation whitelist only accepts GCP. Accept the GCS spelling and keep the validation error consistent with it. GcpTokenProviderTest aborts in EvHttpServer::start because the explicit-host constructor leaves the event-base vector empty. Use the port-only constructor that initializes event bases, as existing HTTP client tests do. ### Release note Accept GCS as a Google Cloud Storage resource provider alias. ### Check List (For Author) - Test: FE Checkstyle, C++ clang-format 16, BE build hygiene and git diff --check pass. - Targeted FE UT blocked by Iceberg dependency TLS failure. - Targeted BE UT blocked by required submodule download; not executed. - Behavior changed: Yes, resource validation accepts GCS as well as GCP. - Does this need documentation: No --- be/test/common/auth/gcp/gcp_token_provider_test.cpp | 3 ++- .../org/apache/doris/datasource/storage/S3ResourceCompat.java | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/be/test/common/auth/gcp/gcp_token_provider_test.cpp b/be/test/common/auth/gcp/gcp_token_provider_test.cpp index c1cc7d9fc10d68..9ce2988e812ba9 100644 --- a/be/test/common/auth/gcp/gcp_token_provider_test.cpp +++ b/be/test/common/auth/gcp/gcp_token_provider_test.cpp @@ -175,7 +175,8 @@ class GcpTokenProviderTest : public testing::Test { std::string private_key; std::string token_uri; OAuthTokenHandler handler; - EvHttpServer server {"127.0.0.1", 0}; + // The port-only constructor initializes the worker event bases before start(). + EvHttpServer server {0}; }; TEST_F(GcpTokenProviderTest, ServiceAccountAdcExchangesAndCachesOAuthToken) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java index 48f26359194792..ebfcddb41faf05 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/S3ResourceCompat.java @@ -57,7 +57,7 @@ public final class S3ResourceCompat { public static final List REQUIRED_FIELDS = Arrays.asList(ENDPOINT); public static final List PROVIDERS = - Arrays.asList("COS", "OSS", "S3", "OBS", "BOS", "AZURE", "GCP", "TOS"); + Arrays.asList("COS", "OSS", "S3", "OBS", "BOS", "AZURE", "GCP", "GCS", "TOS"); private static final Pattern IPV4_PORT_PATTERN = Pattern.compile("((?:\\d{1,3}\\.){3}\\d{1,3}:\\d{1,5})"); @@ -111,7 +111,7 @@ public static void requiredS3Properties(Map properties) throws D if (StringUtils.isNotBlank(properties.get(FS_PROVIDER_KEY))) { // S3 Provider properties should be case insensitive. if (!PROVIDERS.stream().anyMatch(s -> s.equals(properties.get(FS_PROVIDER_KEY).toUpperCase()))) { - throw new DdlException("Provider must be one of OSS, OBS, AZURE, BOS, COS, S3, GCP"); + throw new DdlException("Provider must be one of OSS, OBS, AZURE, BOS, COS, S3, GCP, GCS, TOS"); } } From 67767d4a393962f8245f8d68a238ae4e3f313248 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 23 Oct 2026 12:39:07 +0800 Subject: [PATCH 41/47] [fix](fe) Map GCS alias to GCP in cloud object store adapter ### What problem does this PR solve? Related PR: #68540 Problem Summary: S3 resources accept GCS as an alias for GCP, but the cloud object store adapter passes GCS directly to the protobuf Provider valueOf method. The protocol only defines GCP, so the alias parity test fails with IllegalArgumentException and the corresponding cloud conversion cannot succeed. Map the alias to Provider.GCP, normalize case with Locale.ROOT, and assert the protocol provider in the existing four-alias test. Keep the protobuf schema and other provider validation unchanged. ### Release note Fix cloud object store conversion for the GCS provider alias. ### Check List (For Author) - Test: FE core Maven Checkstyle passed with zero violations; git diff --check passed. Targeted run-fe-ut.sh execution was attempted for S3ResourceTest#testGsAliasesMatchConnectorAndResourceProtocols and CloudObjectStoreAdapterParityTest but stopped before tests because the local Thrift compiler is 0.16.0 while the repository requires 0.24.0. - Behavior changed: Yes; GCS/gcs maps to the existing GCP protocol value. - Does this need documentation: No. --- .../doris/datasource/storage/CloudObjectStoreAdapter.java | 5 ++++- .../test/java/org/apache/doris/catalog/S3ResourceTest.java | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java index a32356112bb088..20a1780308fa36 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/CloudObjectStoreAdapter.java @@ -27,6 +27,7 @@ import com.google.common.base.Preconditions; import com.google.common.base.Strings; +import java.util.Locale; import java.util.Map; import java.util.Optional; @@ -92,7 +93,9 @@ public static Cloud.ObjectStoreInfoPB.Builder getObjStoreInfoPB(Map Date: Fri, 23 Oct 2026 15:09:28 +0800 Subject: [PATCH 42/47] [fix](iceberg) Isolate GCS FileIO settings from Glue clients ### What problem does this PR solve? Related PR: apache/doris#68540 Problem Summary: An Iceberg Glue catalog using native GCP storage overwrote Glue's AWS region with GCS's region. Anonymous GCS storage also replaced Glue's authenticated credential provider. Preserve Glue's metadata settings, install a FileIO-only anonymous client factory, and keep its synchronous and asynchronous clients independent of Glue's static keys or assume-role factory. ### Release note Preserve AWS Glue authentication and region when Iceberg objects use native GCP credentials or anonymous GCS access. ### Check List (For Author) - Test: Checkstyle, connector import boundary, git diff --check, and isolated JDK 17 SDK request/serialization smoke passed. Added three request-level regression tests. Standard FE and connector-reactor tests stopped before these tests at the installed Thrift 0.16.0 versus required 0.24.0 mismatch. No live cloud integration tests were run. - Behavior changed: Yes, Glue keeps its AWS metadata identity and region while GCS FileIO uses the selected storage authentication. - Does this need documentation: No --- .../iceberg/IcebergCatalogFactory.java | 15 +- .../IcebergGlueGcsAuthenticationTest.java | 181 ++++++++++++++++++ .../AnonymousS3FileIOAwsClientFactory.java | 61 ++++++ 3 files changed, 254 insertions(+), 3 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java create mode 100644 fe/fe-iceberg-common/src/main/java/org/apache/doris/connector/iceberg/AnonymousS3FileIOAwsClientFactory.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index bf50308d1763c2..c91543fd0f23e1 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -456,8 +456,13 @@ public static Map buildCatalogProperties(IcebergCatalogPropertie chosenS3.ifPresent(storage -> storage.toBackendProperties().ifPresent(backend -> { Optional auth = GcsAuthResolver.resolve(backend.toMap()); if (auth.filter(GcsAuth::isAnonymous).isPresent()) { - opts.put(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER, - "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider"); + if (IcebergCatalogProperties.TYPE_GLUE.equals(flavor)) { + // Glue metadata still needs AWS credentials when the GCS objects are public. + opts.put(S3FileIOProperties.CLIENT_FACTORY, AnonymousS3FileIOAwsClientFactory.class.getName()); + } else { + opts.put(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER, + "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider"); + } } auth.flatMap(GcsAuth::getNativeCredential).ifPresent(credential -> { // HadoopCatalog resolves its namespace filesystem independently of S3FileIO. @@ -471,7 +476,11 @@ public static Map buildCatalogProperties(IcebergCatalogPropertie opts.put(CatalogProperties.WAREHOUSE_LOCATION, "gs://" + warehouse.substring(6)); } } - putS3FileIODialect(opts, storage); + // Glue already emitted the object-store settings. Its shared client.region belongs + // to the metadata client; the GCP FileIO factory resolves its own storage region. + if (!IcebergCatalogProperties.TYPE_GLUE.equals(flavor)) { + putS3FileIODialect(opts, storage); + } opts.put("provider", "GCP"); opts.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, credential.getCredentialProviderType().name()); putIfNotBlank(opts, GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java new file mode 100644 index 00000000000000..99d7539e08608e --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java @@ -0,0 +1,181 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.iceberg.glue.ConfigurationAWSCredentialsProvider2x; +import org.apache.doris.filesystem.gcs.GcsFileSystemProperties; + +import com.sun.net.httpserver.HttpServer; +import org.apache.iceberg.aws.AwsClientFactories; +import org.apache.iceberg.aws.AwsClientProperties; +import org.apache.iceberg.aws.S3FileIOAwsClientFactories; +import org.apache.iceberg.aws.s3.S3FileIO; +import org.apache.iceberg.aws.s3.S3FileIOAwsClientFactory; +import org.apache.iceberg.aws.s3.S3FileIOProperties; +import org.apache.iceberg.util.SerializationUtil; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import software.amazon.awssdk.services.glue.GlueClient; +import software.amazon.awssdk.services.s3.S3AsyncClient; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +class IcebergGlueGcsAuthenticationTest { + @Test + void nativeGcsPreservesGlueRegionAndSigningCredentials() throws Exception { + try (RequestRecorder recorder = new RequestRecorder()) { + for (String mode : List.of("DEFAULT", "COMPUTE_ENGINE")) { + for (String account : List.of("", "reader@test.iam.gserviceaccount.com")) { + Map raw = properties(mode, recorder.endpoint()); + raw.put("gs.impersonation_service_account", account); + Map options = catalogOptions(raw); + Assertions.assertEquals("us-west-2", options.get(AwsClientProperties.CLIENT_REGION)); + Assertions.assertEquals(GcpS3FileIOAwsClientFactory.class.getName(), + options.get(S3FileIOProperties.CLIENT_FACTORY)); + assertGlueRequestIsSigned(options, recorder); + } + } + Assertions.assertEquals(4, recorder.glueRequests.get()); + } + } + + @Test + void anonymousGcsPreservesGlueCredentialsAndSendsUnsignedObjectRequests() throws Exception { + try (RequestRecorder recorder = new RequestRecorder()) { + Map options = catalogOptions(properties("ANONYMOUS", recorder.endpoint())); + Assertions.assertEquals(ConfigurationAWSCredentialsProvider2x.class.getName(), + options.get(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER)); + assertGlueRequestIsSigned(options, recorder); + assertObjectRequestsAreUnsigned(options, recorder); + // Initializing FileIO must leave the catalog's credential provider intact. + assertGlueRequestIsSigned(options, recorder); + } + } + + @Test + void anonymousGcsFileIoDoesNotAssumeTheGlueRole() throws Exception { + try (RequestRecorder recorder = new RequestRecorder()) { + Map raw = properties("ANONYMOUS", recorder.endpoint()); + raw.remove("glue.access_key"); + raw.remove("glue.secret_key"); + raw.put("glue.role_arn", "arn:aws:iam::123456789012:role/glue-reader"); + Map options = catalogOptions(raw); + Assertions.assertEquals("org.apache.iceberg.aws.AssumeRoleAwsClientFactory", + options.get("client.factory")); + assertObjectRequestsAreUnsigned(options, recorder); + Assertions.assertEquals(0, recorder.glueRequests.get()); + } + } + + private static Map properties(String mode, String endpoint) { + Map raw = new HashMap<>(Map.of( + "iceberg.catalog.type", "glue", "provider", "GCP", + "warehouse", "gs://bucket/warehouse", "gs.credential_provider_type", mode, + "glue.access_key", "glue-test-ak", "glue.secret_key", "glue-test-sk", + "glue.region", "us-west-2", "glue.endpoint", endpoint)); + if ("ANONYMOUS".equals(mode)) { + raw.put("gs.endpoint", endpoint); + raw.put("use_path_style", "true"); + // Exercise the Java async client without the optional native CRT dependency. + raw.put(S3FileIOProperties.S3_CRT_ENABLED, "false"); + } + return raw; + } + + private static Map catalogOptions(Map raw) { + return IcebergCatalogFactory.buildCatalogProperties(IcebergCatalogProperties.of(raw), + Optional.of(GcsFileSystemProperties.of(raw)), false); + } + + private static void assertGlueRequestIsSigned(Map options, RequestRecorder recorder) { + try (GlueClient glue = AwsClientFactories.from(options).glue()) { + Assertions.assertTrue(glue.getDatabases(b -> { }).databaseList().isEmpty()); + Assertions.assertEquals("us-west-2", glue.serviceClientConfiguration().region().id()); + } + String authorization = recorder.glueAuthorization.get(); + Assertions.assertNotNull(authorization); + Assertions.assertTrue(authorization.contains("Credential=glue-test-ak/"), authorization); + Assertions.assertTrue(authorization.contains("/us-west-2/glue/aws4_request"), authorization); + } + + private static void assertObjectRequestsAreUnsigned(Map options, RequestRecorder recorder) { + try (S3FileIO io = new S3FileIO()) { + io.initialize(options); + Assertions.assertEquals(8L, io.newInputFile("s3://bucket/object").getLength()); + } + Assertions.assertEquals(1, recorder.objectRequests.get()); + Assertions.assertNull(recorder.objectAuthorization.get()); + S3FileIOAwsClientFactory factory = S3FileIOAwsClientFactories.initialize(options); + factory = SerializationUtil.deserializeFromBytes(SerializationUtil.serializeToBytes(factory)); + try (S3AsyncClient client = factory.s3Async()) { + Assertions.assertEquals(8L, client.headObject(b -> b.bucket("bucket").key("object")) + .join().contentLength()); + } + Assertions.assertEquals(2, recorder.objectRequests.get()); + Assertions.assertNull(recorder.objectAuthorization.get()); + } + + private static final class RequestRecorder implements AutoCloseable { + private final HttpServer server; + private final AtomicReference glueAuthorization = new AtomicReference<>(); + private final AtomicReference objectAuthorization = new AtomicReference<>(); + private final AtomicInteger glueRequests = new AtomicInteger(); + private final AtomicInteger objectRequests = new AtomicInteger(); + + private RequestRecorder() throws IOException { + server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext("/", exchange -> { + try (exchange) { + exchange.getRequestBody().readAllBytes(); + if ("HEAD".equals(exchange.getRequestMethod())) { + objectRequests.incrementAndGet(); + objectAuthorization.set(exchange.getRequestHeaders().getFirst("Authorization")); + exchange.getResponseHeaders().set("Content-Length", "8"); + exchange.sendResponseHeaders(200, -1); + } else { + glueRequests.incrementAndGet(); + glueAuthorization.set(exchange.getRequestHeaders().getFirst("Authorization")); + byte[] response = "{\"DatabaseList\":[]}".getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().set("Content-Type", "application/x-amz-json-1.1"); + exchange.sendResponseHeaders(200, response.length); + exchange.getResponseBody().write(response); + } + } + }); + server.start(); + } + + private String endpoint() { + return "http://127.0.0.1:" + server.getAddress().getPort(); + } + + @Override + public void close() { + server.stop(0); + } + } + +} diff --git a/fe/fe-iceberg-common/src/main/java/org/apache/doris/connector/iceberg/AnonymousS3FileIOAwsClientFactory.java b/fe/fe-iceberg-common/src/main/java/org/apache/doris/connector/iceberg/AnonymousS3FileIOAwsClientFactory.java new file mode 100644 index 00000000000000..25f027016743fe --- /dev/null +++ b/fe/fe-iceberg-common/src/main/java/org/apache/doris/connector/iceberg/AnonymousS3FileIOAwsClientFactory.java @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.iceberg.aws.AwsClientFactories; +import org.apache.iceberg.aws.AwsClientFactory; +import org.apache.iceberg.aws.AwsClientProperties; +import org.apache.iceberg.aws.AwsProperties; +import org.apache.iceberg.aws.s3.S3FileIOAwsClientFactory; +import org.apache.iceberg.aws.s3.S3FileIOProperties; +import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider; +import software.amazon.awssdk.services.s3.S3AsyncClient; +import software.amazon.awssdk.services.s3.S3Client; + +import java.util.HashMap; +import java.util.Map; + +/** Keeps anonymous object access separate from the catalog's authenticated AWS metadata clients. */ +public class AnonymousS3FileIOAwsClientFactory implements S3FileIOAwsClientFactory { + private static final long serialVersionUID = 1L; + + private AwsClientFactory delegate; + + @Override + public void initialize(Map properties) { + Map fileIoProperties = new HashMap<>(properties); + // A Glue assume-role factory or explicit object keys must not override anonymous FileIO. + fileIoProperties.remove(AwsProperties.CLIENT_FACTORY); + fileIoProperties.remove(S3FileIOProperties.ACCESS_KEY_ID); + fileIoProperties.remove(S3FileIOProperties.SECRET_ACCESS_KEY); + fileIoProperties.remove(S3FileIOProperties.SESSION_TOKEN); + fileIoProperties.put(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER, + AnonymousCredentialsProvider.class.getName()); + delegate = AwsClientFactories.from(fileIoProperties); + } + + @Override + public S3Client s3() { + return delegate.s3(); + } + + @Override + public S3AsyncClient s3Async() { + return delegate.s3Async(); + } +} From 33dc40dbc201aac19402ab15b2076305fd4da47a Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 23 Oct 2026 16:41:02 +0800 Subject: [PATCH 43/47] [fix](iceberg) Separate metadata credentials from GCS object authentication ### What problem does this PR solve? Related PR: #68540 Problem Summary: GCS property validation incorrectly treats Glue roles and Glue/REST credential-provider selectors as object-storage authentication. This rejects otherwise independent metadata settings and prevents the Glue-role FileIO test from reaching client construction. Add a storage credential view for validation and canonical credential selection while retaining the original raw properties and rejecting AWS storage conflicts. Anonymous GCS also overwrites the shared credential provider used for REST SigV4 authentication. Configure the existing anonymous S3 FileIO factory for every catalog flavor, preserving metadata client credentials. Add coverage for metadata-scoped options across native, anonymous and HMAC GCS modes, REST default/selected providers, and signed REST requests alongside unsigned object requests. Update the existing anonymous FileIO expectation and use URLConnection for local transport tests. ### Release note Fix GCS catalogs rejecting independent Glue authentication settings and anonymous GCS object access disabling REST catalog SigV4 authentication. ### Check List (For Author) - Test: GCS reactor tests passed, including 56 GCS and 152 S3-base tests; Checkstyle passed for all affected modules with zero violations; connector import and metadata-funnel gates and git diff --check passed. An isolated Iceberg SDK transport probe passed signed REST and unsigned object requests plus default/selected metadata provider checks. That probe uses expected SDK options and does not replace the connector suite. Full connector tests were attempted but blocked by Maven dependency TLS failures while downloading Jackson 2.21.8, before connector test execution. - Behavior changed: Yes; metadata authentication remains independent of GCS. - Does this need documentation: No --- .../iceberg/IcebergCatalogFactory.java | 9 +- .../iceberg/IcebergCatalogFactoryTest.java | 6 +- .../IcebergGlueGcsAuthenticationTest.java | 3 + .../IcebergRestGcsAuthenticationTest.java | 149 ++++++++++++++++++ .../gcs/GcsFileSystemProperties.java | 11 ++ .../gcs/GcsFileSystemPropertiesTest.java | 42 +++++ .../s3/AbstractDelegatingS3Properties.java | 9 +- 7 files changed, 217 insertions(+), 12 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestGcsAuthenticationTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index c91543fd0f23e1..4f75779e494c51 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -456,13 +456,8 @@ public static Map buildCatalogProperties(IcebergCatalogPropertie chosenS3.ifPresent(storage -> storage.toBackendProperties().ifPresent(backend -> { Optional auth = GcsAuthResolver.resolve(backend.toMap()); if (auth.filter(GcsAuth::isAnonymous).isPresent()) { - if (IcebergCatalogProperties.TYPE_GLUE.equals(flavor)) { - // Glue metadata still needs AWS credentials when the GCS objects are public. - opts.put(S3FileIOProperties.CLIENT_FACTORY, AnonymousS3FileIOAwsClientFactory.class.getName()); - } else { - opts.put(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER, - "software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider"); - } + // Glue and REST metadata can require AWS authentication even when GCS objects are public. + opts.put(S3FileIOProperties.CLIENT_FACTORY, AnonymousS3FileIOAwsClientFactory.class.getName()); } auth.flatMap(GcsAuth::getNativeCredential).ifPresent(credential -> { // HadoopCatalog resolves its namespace filesystem independently of S3FileIO. diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index e3d99e0d6d0631..4fae7f3286e324 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -1234,8 +1234,8 @@ void anonymousGcpCatalogUsesUnsignedFileIo() { Map options = IcebergCatalogFactory.buildCatalogProperties( IcebergCatalogProperties.of(raw), Optional.of(GcsFileSystemProperties.of(raw)), false); - Assertions.assertEquals(AnonymousCredentialsProvider.class.getName(), - options.get(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER)); - Assertions.assertFalse(options.containsKey("s3.client-factory")); + Assertions.assertEquals(AnonymousS3FileIOAwsClientFactory.class.getName(), + options.get(S3FileIOProperties.CLIENT_FACTORY)); + Assertions.assertFalse(options.containsKey(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER)); } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java index 99d7539e08608e..ccf5775a013c27 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergGlueGcsAuthenticationTest.java @@ -23,6 +23,7 @@ import com.sun.net.httpserver.HttpServer; import org.apache.iceberg.aws.AwsClientFactories; import org.apache.iceberg.aws.AwsClientProperties; +import org.apache.iceberg.aws.HttpClientProperties; import org.apache.iceberg.aws.S3FileIOAwsClientFactories; import org.apache.iceberg.aws.s3.S3FileIO; import org.apache.iceberg.aws.s3.S3FileIOAwsClientFactory; @@ -96,6 +97,8 @@ private static Map properties(String mode, String endpoint) { "warehouse", "gs://bucket/warehouse", "gs.credential_provider_type", mode, "glue.access_key", "glue-test-ak", "glue.secret_key", "glue-test-sk", "glue.region", "us-west-2", "glue.endpoint", endpoint)); + // Keep the local transport test independent of the optional Apache HTTP client. + raw.put(HttpClientProperties.CLIENT_TYPE, HttpClientProperties.CLIENT_TYPE_URLCONNECTION); if ("ANONYMOUS".equals(mode)) { raw.put("gs.endpoint", endpoint); raw.put("use_path_style", "true"); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestGcsAuthenticationTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestGcsAuthenticationTest.java new file mode 100644 index 00000000000000..632ce72c06d83f --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRestGcsAuthenticationTest.java @@ -0,0 +1,149 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.filesystem.gcs.GcsFileSystemProperties; + +import com.sun.net.httpserver.HttpServer; +import org.apache.iceberg.aws.AwsClientProperties; +import org.apache.iceberg.aws.AwsProperties; +import org.apache.iceberg.aws.HttpClientProperties; +import org.apache.iceberg.aws.s3.S3FileIO; +import org.apache.iceberg.aws.s3.S3FileIOProperties; +import org.apache.iceberg.rest.RESTCatalog; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.AwsCredentials; +import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider; + +import java.net.InetSocketAddress; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +class IcebergRestGcsAuthenticationTest { + @Test + void anonymousGcsPreservesRestDefaultAndSelectedCredentialProviders() { + Map raw = properties("http://127.0.0.1"); + Map options = catalogOptions(raw); + Assertions.assertFalse(options.containsKey(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER)); + Assertions.assertInstanceOf(DefaultCredentialsProvider.class, + new AwsProperties(options).restCredentialsProvider()); + raw.put("iceberg.rest.credentials_provider_type", "SYSTEM_PROPERTIES"); + options = catalogOptions(raw); + Assertions.assertEquals(SystemPropertyCredentialsProvider.class.getName(), + options.get(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER)); + Assertions.assertInstanceOf(SystemPropertyCredentialsProvider.class, + new AwsProperties(options).restCredentialsProvider()); + Assertions.assertEquals(AnonymousS3FileIOAwsClientFactory.class.getName(), + options.get(S3FileIOProperties.CLIENT_FACTORY)); + } + + @Test + void anonymousGcsObjectsDoNotDisableRestRequestSigning() throws Exception { + AtomicReference restAuthorization = new AtomicReference<>(); + AtomicReference objectAuthorization = new AtomicReference<>(); + AtomicInteger restRequests = new AtomicInteger(); + AtomicInteger objectRequests = new AtomicInteger(); + HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext("/", exchange -> { + try (exchange) { + exchange.getRequestBody().readAllBytes(); + String authorization = exchange.getRequestHeaders().getFirst("Authorization"); + if ("HEAD".equals(exchange.getRequestMethod())) { + objectRequests.incrementAndGet(); + objectAuthorization.set(authorization); + exchange.getResponseHeaders().set("Content-Length", "8"); + exchange.sendResponseHeaders(200, -1); + } else { + restRequests.incrementAndGet(); + restAuthorization.set(authorization); + byte[] response = "{\"defaults\":{},\"overrides\":{}}".getBytes(StandardCharsets.UTF_8); + exchange.getResponseHeaders().set("Content-Type", "application/json"); + exchange.sendResponseHeaders(200, response.length); + exchange.getResponseBody().write(response); + } + } + }); + server.start(); + try { + String endpoint = "http://127.0.0.1:" + server.getAddress().getPort(); + Map raw = properties(endpoint); + raw.put(AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER, MetadataCredentialsProvider.class.getName()); + Map options = catalogOptions(raw); + assertRestRequestIsSigned(options, restAuthorization); + try (S3FileIO io = new S3FileIO()) { + io.initialize(options); + Assertions.assertEquals(8L, io.newInputFile("s3://bucket/object").getLength()); + } + Assertions.assertEquals(1, objectRequests.get()); + Assertions.assertNull(objectAuthorization.get()); + // Creating an object client must not mutate the REST catalog's credential configuration. + assertRestRequestIsSigned(options, restAuthorization); + Assertions.assertTrue(restRequests.get() >= 2); + } finally { + server.stop(0); + } + } + + private static void assertRestRequestIsSigned(Map options, + AtomicReference authorization) throws Exception { + authorization.set(null); + try (RESTCatalog catalog = new RESTCatalog()) { + catalog.initialize("rest-gcs", options); + } + String header = authorization.get(); + Assertions.assertNotNull(header); + Assertions.assertTrue(header.contains("Credential=rest-test-ak/"), header); + Assertions.assertTrue(header.contains("/us-west-2/execute-api/aws4_request"), header); + } + + private static Map properties(String endpoint) { + Map raw = new HashMap<>(Map.of( + "iceberg.catalog.type", "rest", "iceberg.rest.uri", endpoint, + "iceberg.rest.signing-name", "execute-api", "iceberg.rest.sigv4-enabled", "true", + "iceberg.rest.signing-region", "us-west-2", "provider", "GCP", + "gs.credential_provider_type", "ANONYMOUS", "gs.endpoint", endpoint, + "use_path_style", "true", S3FileIOProperties.S3_CRT_ENABLED, "false")); + // Keep the local transport test independent of the optional Apache HTTP client. + raw.put(HttpClientProperties.CLIENT_TYPE, HttpClientProperties.CLIENT_TYPE_URLCONNECTION); + return raw; + } + + private static Map catalogOptions(Map raw) { + return IcebergCatalogFactory.buildCatalogProperties(IcebergCatalogProperties.of(raw), + Optional.of(GcsFileSystemProperties.of(raw)), false); + } + + public static final class MetadataCredentialsProvider implements AwsCredentialsProvider { + public static MetadataCredentialsProvider create() { + return new MetadataCredentialsProvider(); + } + + @Override + public AwsCredentials resolveCredentials() { + return AwsBasicCredentials.create("rest-test-ak", "rest-test-sk"); + } + } +} diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java index 74df1420679eb8..513b68127a0a64 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java @@ -132,6 +132,17 @@ public static GcsFileSystemProperties of(Map properties) { return props; } + @Override + protected Map credentialProperties() { + // Unlike S3's legacy aliases, GCS never uses Glue/REST credentials for object access. + Map storage = new HashMap<>(rawProperties()); + storage.remove("glue.role_arn"); + storage.remove("glue.external_id"); + storage.remove("glue.credentials_provider_type"); + storage.remove("iceberg.rest.credentials_provider_type"); + return storage; + } + @Override public void validate() { validateHmacDialect("gs", "GCS"); diff --git a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java index d4d0cdfa60aed5..9b5d92b01a56ec 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java @@ -217,6 +217,48 @@ void validate_rejectsAwsOnlyCredentialOptions() { Assertions.assertTrue(e.getMessage().contains("HMAC"), e.getMessage()); } + @Test + void metadataCredentialsDoNotSelectOrInvalidateGcsAuthentication() { + Map metadata = Map.of( + "glue.role_arn", "arn:aws:iam::1:role/metadata", + "glue.external_id", "metadata-external-id", + "glue.credentials_provider_type", "INSTANCE_PROFILE", + "iceberg.rest.credentials_provider_type", "SYSTEM_PROPERTIES"); + for (String mode : List.of("DEFAULT", "COMPUTE_ENGINE", "ANONYMOUS", "HMAC")) { + Map raw = new HashMap<>(metadata); + raw.put("provider", "GCP"); + if ("HMAC".equals(mode)) { + raw.put("gs.access_key", "gcs-ak"); + raw.put("gs.secret_key", "gcs-sk"); + } else { + raw.put("gs.credential_provider_type", mode); + } + GcsFileSystemProperties properties = GcsFileSystemProperties.of(raw); + Map backend = properties.toMap(); + Assertions.assertEquals(raw, properties.rawProperties()); + if ("ANONYMOUS".equals(mode)) { + Assertions.assertEquals("ANONYMOUS", backend.get("AWS_CREDENTIALS_PROVIDER_TYPE")); + } else { + Assertions.assertFalse(backend.containsKey("AWS_CREDENTIALS_PROVIDER_TYPE")); + } + Assertions.assertEquals("", properties.getRoleArn()); + Assertions.assertEquals("", properties.getExternalId()); + for (String key : metadata.keySet()) { + Assertions.assertFalse(backend.containsKey(key), key); + } + // Actual AWS storage credentials must still conflict with GCS authentication. + for (Map.Entry credential : Map.of( + "s3.role_arn", "arn:aws:iam::1:role/storage", + "sts.role_arn", "arn:aws:iam::1:role/storage", + "s3.credentials_provider_type", "INSTANCE_PROFILE").entrySet()) { + Map conflicting = new HashMap<>(raw); + conflicting.put(credential.getKey(), credential.getValue()); + Assertions.assertThrows(IllegalArgumentException.class, + () -> GcsFileSystemProperties.of(conflicting), mode + ": " + credential.getKey()); + } + } + } + @Test void toString_masksSecretsButNotAccessKey() { String rendered = GcsFileSystemProperties.of(Map.of( diff --git a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/AbstractDelegatingS3Properties.java b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/AbstractDelegatingS3Properties.java index 3e502a2828d008..a95fc0bd45be4d 100644 --- a/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/AbstractDelegatingS3Properties.java +++ b/fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/AbstractDelegatingS3Properties.java @@ -106,12 +106,17 @@ protected final void validateHmacDialect(String aliasPrefix, String displayName) "Property " + aliasPrefix + ".endpoint is required.") .check(this::hasInvalidUsePathStyle, "use_path_style must be true or false, got: '" + getUsePathStyle() + "'") - .check(() -> S3CompatSignals.hasAwsOnlyCredentialOptions(rawProperties), + .check(() -> S3CompatSignals.hasAwsOnlyCredentialOptions(credentialProperties()), displayName + " supports only HMAC access_key/secret_key credentials; " + "role_arn/instance-profile style options are not supported") .validate("Invalid " + displayName + " filesystem properties"); } + /** Raw credential aliases belonging to this storage, excluding independent metadata clients. */ + protected Map credentialProperties() { + return rawProperties; + } + @Override public StorageKind kind() { return StorageKind.OBJECT_STORAGE; @@ -175,7 +180,7 @@ public final Map toS3CompatibleKv() { // requested mode (validation restricts it to DEFAULT/ANONYMOUS): a user asking for DEFAULT // keeps the credential chain instead of being silently downgraded to unsigned ANONYMOUS. if (!hasStaticCredentials()) { - String requested = S3CompatSignals.requestedCredentialsProviderType(rawProperties); + String requested = S3CompatSignals.requestedCredentialsProviderType(credentialProperties()); kv.put("AWS_CREDENTIALS_PROVIDER_TYPE", requested != null ? requested : "ANONYMOUS"); } customizeS3CompatibleKv(kv); From 0505a9fd63ee0926bda7107909ebc5d111f43bd8 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 23 Oct 2026 17:33:21 +0800 Subject: [PATCH 44/47] [fix](storage) Preserve legacy GCS static key precedence ### What problem does this PR solve? Related PR: #68540 Problem Summary: Older FEs accepted GCS resources containing static HMAC keys and a dormant legacy ANONYMOUS credential selector. Resolving the selector before the keys rejected those persisted resources after upgrade and aborted batched storage policy updates. Resolve legacy static keys first while keeping explicit native GCP authentication conflicts rejected. Cover credential aliases, partial patches, metadata replay, and batched resource/policy updates and drops. ### Release note Preserve access to legacy GCS resources with static keys and an anonymous selector when upgrading to native GCP authentication support. ### Check List (For Author) - Test: A 192-case source probe fails before the fix and passes after it; Java parsing, FE Core/GCS Checkstyle, and git diff --check passed. Added resolver and replay/batched-push unit tests. The standard FE UT runner stopped during Thrift code generation with the installed 0.16.0 compiler; FE tests were not executed. - Behavior changed: Yes, restore legacy static-key precedence without weakening explicit native GCP credential conflict validation. - Does this need documentation: No --- .../apache/doris/catalog/S3ResourceTest.java | 63 +++++++++++++++++++ .../filesystem/gcs/auth/GcsAuthResolver.java | 5 +- .../gcs/auth/GcsAuthResolverTest.java | 35 +++++++++++ 3 files changed, 102 insertions(+), 1 deletion(-) diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 02748c778191f9..476a2c76c4fa3b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -32,6 +32,7 @@ import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.commands.CreateResourceCommand; import org.apache.doris.nereids.trees.plans.commands.info.CreateResourceInfo; +import org.apache.doris.policy.StoragePolicy; import org.apache.doris.qe.ConnectContext; import org.apache.doris.task.PushStoragePolicyTask; import org.apache.doris.thrift.TGcpCredential; @@ -708,6 +709,68 @@ public void testLegacyGcpDefaultChainSurvivesReplayAndPolicyPush() throws Except } } + @Test + public void testLegacyGcpStaticKeysSurviveReplayAndBatchedPolicyPush() throws Exception { + for (String key : new String[] {"s3.credentials_provider_type", "AWS_CREDENTIALS_PROVIDER_TYPE"}) { + Map properties = new HashMap<>(s3Properties); + properties.put("provider", "GCP"); + S3Resource resource = new S3Resource("legacy_gcp_static_keys"); + resource.id = 1L; + resource.version = 1L; + resource.setProperties(ImmutableMap.copyOf(properties)); + // Recreate metadata accepted by older FEs without invoking current CREATE validation. + Map legacyProperties = resource.getCopiedProperties(); + legacyProperties.put(key, "ANONYMOUS"); + Field field = S3Resource.class.getDeclaredField("properties"); + field.setAccessible(true); + field.set(resource, legacyProperties); + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (DataOutputStream output = new DataOutputStream(bytes)) { + resource.write(output); + } + S3Resource restored; + try (DataInputStream input = new DataInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + restored = (S3Resource) Resource.read(input); + } + ResourceMgr resourceMgr = new ResourceMgr(); + resourceMgr.replayCreateResource(restored); + Assertions.assertEquals(legacyProperties, restored.getCopiedProperties()); + Assertions.assertSame(restored, resourceMgr.getResource(restored.getName())); + + S3Resource unrelated = new S3Resource("unrelated_resource"); + unrelated.id = 2L; + unrelated.version = 1L; + unrelated.setProperties(ImmutableMap.copyOf(s3Properties)); + resourceMgr.replayCreateResource(unrelated); + StoragePolicy policy = Mockito.mock(StoragePolicy.class); + Mockito.when(policy.getId()).thenReturn(3L); + Mockito.when(policy.getPolicyName()).thenReturn("unrelated_policy"); + Mockito.when(policy.getStorageResource()).thenReturn(unrelated.getName()); + try (MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + Env env = Mockito.mock(Env.class); + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + Mockito.when(env.getResourceMgr()).thenReturn(resourceMgr); + PushStoragePolicyTask task = new PushStoragePolicyTask(1L, List.of(policy), + List.of(restored, unrelated), List.of(9L)); + TPushStoragePolicyReq request = task.toThrift(); + Assertions.assertEquals(2, request.getResourceSize()); + Assertions.assertEquals(1L, request.getResource().get(0).getId()); + Assertions.assertEquals(2L, request.getResource().get(1).getId()); + Assertions.assertEquals(1, request.getStoragePolicySize()); + Assertions.assertEquals(3L, request.getStoragePolicy().get(0).getId()); + Assertions.assertEquals(2L, request.getStoragePolicy().get(0).getResourceId()); + Assertions.assertEquals(List.of(9L), request.getDroppedStoragePolicy()); + TS3StorageParam thrift = request.getResource().get(0).getS3StorageParam(); + Assertions.assertEquals(TObjStorageType.GCP, thrift.getProvider()); + Assertions.assertEquals(s3AccessKey, thrift.getAk()); + Assertions.assertEquals(s3SecretKey, thrift.getSk()); + Assertions.assertFalse(thrift.isSetCredential()); + Assertions.assertFalse(thrift.isSetCredProviderType()); + Assertions.assertEquals(s3AccessKey, request.getResource().get(1).getS3StorageParam().getAk()); + } + } + } + @Test public void testMinioAlterSurvivesReplayAndBatchedPolicyPush() throws Exception { S3Resource minio = new S3Resource("minio_resource"); diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java index 6b748b40336bb4..dce0d7729bf04b 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/main/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolver.java @@ -71,9 +71,12 @@ public static Optional resolve(Map properties) { : GcpCredentialProviderType.fromString(CREDENTIAL_PROVIDER_TYPE, providerType); mode = type == GcpCredentialProviderType.ANONYMOUS ? GcsAuth.Mode.ANONYMOUS : type == GcpCredentialProviderType.COMPUTE_ENGINE ? GcsAuth.Mode.COMPUTE_ENGINE : GcsAuth.Mode.ADC; + } else if (hasAccessKey || hasSecretKey) { + // Legacy static credentials take precedence over a dormant anonymous selector. + mode = GcsAuth.Mode.HMAC; } else if (hasLegacyAnonymous) { mode = GcsAuth.Mode.ANONYMOUS; - } else if (hasAccessKey || hasSecretKey || hasLegacyProvider(properties, "DEFAULT")) { + } else if (hasLegacyProvider(properties, "DEFAULT")) { // Explicit legacy DEFAULT uses ambient HMAC credentials through the AWS default chain. mode = GcsAuth.Mode.HMAC; } else { diff --git a/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java b/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java index f056848fee097c..9b6a39f9e658c5 100644 --- a/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java +++ b/fe/fe-filesystem/fe-filesystem-gcs-common/src/test/java/org/apache/doris/filesystem/gcs/auth/GcsAuthResolverTest.java @@ -124,6 +124,41 @@ public void testLegacyGcpDefaultChainDoesNotAcquireAdc() { } } + @Test + public void testLegacyStaticKeysTakePrecedenceOverAnonymous() { + for (String selector : new String[] {"s3.credentials_provider_type", "AWS_CREDENTIALS_PROVIDER_TYPE"}) { + for (String[] keys : new String[][] {{"gs.access_key", "gs.secret_key"}, + {"s3.access_key", "s3.secret_key"}, {"AWS_ACCESS_KEY", "AWS_SECRET_KEY"}, + {"ACCESS_KEY", "SECRET_KEY"}}) { + Map props = new HashMap<>(); + props.put("provider", "GCP"); + props.put(selector, " anonymous "); + props.put(keys[0], "ak"); + props.put(keys[1], "sk"); + GcsAuth auth = GcsAuthResolver.resolve(props).orElseThrow(); + Assertions.assertEquals(GcsAuth.Mode.HMAC, auth.getMode()); + Assertions.assertFalse(auth.isAnonymous()); + Assertions.assertTrue(auth.getNativeCredential().isEmpty()); + for (String mode : new String[] {"DEFAULT", "COMPUTE_ENGINE", "ANONYMOUS"}) { + props.put(GcpCredential.CREDENTIAL_PROVIDER_TYPE, mode); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + } + props.remove(GcpCredential.CREDENTIAL_PROVIDER_TYPE); + props.put(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT, "target@test.iam.gserviceaccount.com"); + Assertions.assertThrows(IllegalArgumentException.class, () -> GcsAuthResolver.resolve(props)); + props.remove(GcpCredential.IMPERSONATION_SERVICE_ACCOUNT); + // Partial vault patches must also retain HMAC authentication. + props.remove(keys[1]); + Assertions.assertEquals(GcsAuth.Mode.HMAC, GcsAuthResolver.resolve(props).orElseThrow().getMode()); + props.remove(keys[0]); + props.put(keys[1], "sk"); + Assertions.assertEquals(GcsAuth.Mode.HMAC, GcsAuthResolver.resolve(props).orElseThrow().getMode()); + props.remove(keys[1]); + Assertions.assertTrue(GcsAuthResolver.resolve(props).orElseThrow().isAnonymous()); + } + } + } + @Test public void testOtherProvidersAndConflictingNativeProperties() { for (String provider : new String[] {"S3", "AZURE", "OSS"}) { From 7d62677930dd873a97fe164c49069b276bdd9555 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 23 Oct 2026 21:16:06 +0800 Subject: [PATCH 45/47] [fix](storage) Preserve resource and vault ALTER patch semantics ### What problem does this PR solve? Related PR: #68540 Problem Summary: Resource updates must validate the effective configuration before publishing it, and concurrent ALTER requests must not overwrite each other's snapshots. Vault ALTER must distinguish absent fields from explicit clears, preserve the existing AWS credential provider on role-only updates, and leave both vault and instance names unchanged after a rejected request. Extract these fixes and their AWS/static-key regression coverage from #68540. Validate HDFS vault properties before filesystem selection as well. This prerequisite introduces no GCP SDK, native credential type, or wire field. ### Release note Resource ALTER serializes updates and publishes only validated configuration. Vault ALTER preserves omitted credentials and rejects conflicting effective authentication without persisting a partial rename. Clearing an unused AWS role preserves static keys and encryption metadata. ### Check List (For Author) - Test: FE Core Checkstyle, Java parsing, clang-format 16 checks on the four changed C++ files, and git diff --check passed. Added concurrency, request field-presence, encrypted-key preservation, partial AWS updates, and failed rename regression coverage. Standard FE UT stopped at the installed Thrift 0.16.0 versus required 0.24.0 check. The standard Cloud build stopped during thirdparty setup because LLVM 20 is unavailable. Unit tests did not execute. - Behavior changed: Yes, effective resource/vault updates preserve omitted state and failed updates do not publish partial metadata. - Does this need documentation: No new properties or deployment requirements. --- cloud/src/common/CMakeLists.txt | 1 + cloud/src/common/auth/obj_credential.cpp | 47 +++++ cloud/src/common/auth/obj_credential.h | 30 +++ .../meta-service/meta_service_resource.cpp | 79 +++++-- cloud/test/meta_service_test.cpp | 195 ++++++++++++++++++ .../doris/catalog/HdfsStorageVault.java | 38 ++-- .../org/apache/doris/catalog/S3Resource.java | 95 +++++---- .../apache/doris/catalog/StorageVaultMgr.java | 55 +++-- .../apache/doris/catalog/S3ResourceTest.java | 67 ++++++ .../cloud/catalog/HdfsStorageVaultTest.java | 45 ++++ .../cloud/catalog/S3StorageVaultTest.java | 126 +++++++++++ 11 files changed, 687 insertions(+), 91 deletions(-) create mode 100644 cloud/src/common/auth/obj_credential.cpp create mode 100644 cloud/src/common/auth/obj_credential.h create mode 100644 fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java diff --git a/cloud/src/common/CMakeLists.txt b/cloud/src/common/CMakeLists.txt index 9f29e1a183b0a3..e356861dd728d9 100644 --- a/cloud/src/common/CMakeLists.txt +++ b/cloud/src/common/CMakeLists.txt @@ -32,6 +32,7 @@ set(COMMON_FILES version_metrics.cpp kms.cpp network_util.cpp + auth/obj_credential.cpp ../server/oss/cloud_server_starter_factory.cpp ) diff --git a/cloud/src/common/auth/obj_credential.cpp b/cloud/src/common/auth/obj_credential.cpp new file mode 100644 index 00000000000000..15c6e61345357c --- /dev/null +++ b/cloud/src/common/auth/obj_credential.cpp @@ -0,0 +1,47 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "common/auth/obj_credential.h" + +#include + +namespace doris::cloud { + +std::optional validate_obj_authentication(const ObjectStoreInfoPB& obj) { + if (obj.has_role_arn() && obj.role_arn().empty()) { + return "AWS role ARN cannot be empty"; + } + bool has_static_credentials = + !obj.ak().empty() || !obj.sk().empty() || obj.has_encryption_info(); + bool has_aws_authentication = + !obj.role_arn().empty() || !obj.external_id().empty() || obj.has_cred_provider_type(); + if (has_static_credentials && has_aws_authentication) { + return "access keys cannot be combined with AWS role or credential provider"; + } + if (!has_aws_authentication) { + return std::nullopt; + } + if (!obj.has_provider() || obj.provider() != ObjectStoreInfoPB::S3) { + return "AWS role and credential provider require provider=S3"; + } + if (!obj.external_id().empty() && obj.role_arn().empty()) { + return "AWS external ID requires a role ARN"; + } + return std::nullopt; +} + +} // namespace doris::cloud diff --git a/cloud/src/common/auth/obj_credential.h b/cloud/src/common/auth/obj_credential.h new file mode 100644 index 00000000000000..a4107e971376ec --- /dev/null +++ b/cloud/src/common/auth/obj_credential.h @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include + +namespace doris::cloud { + +class ObjectStoreInfoPB; + +// Validate the complete persisted authentication state after applying an ALTER patch. +std::optional validate_obj_authentication(const ObjectStoreInfoPB& obj); + +} // namespace doris::cloud diff --git a/cloud/src/meta-service/meta_service_resource.cpp b/cloud/src/meta-service/meta_service_resource.cpp index 0284d4246efcc1..49637d677a4914 100644 --- a/cloud/src/meta-service/meta_service_resource.cpp +++ b/cloud/src/meta-service/meta_service_resource.cpp @@ -31,6 +31,7 @@ #include #include +#include "common/auth/obj_credential.h" #include "common/bvars.h" #include "common/config.h" #include "common/encryption_util.h" @@ -782,13 +783,16 @@ static void create_object_info_with_encrypt(const InstanceInfoPB& instance, Obje std::string region = obj->has_region() ? obj->region() : ""; if (obj->has_role_arn()) { - if (obj->role_arn().empty() || !obj->has_cred_provider_type() || !obj->has_provider() || + if (obj->role_arn().empty() || !obj->has_provider() || obj->provider() != ObjectStoreInfoPB::S3 || bucket.empty() || endpoint.empty() || region.empty()) { code = MetaServiceCode::INVALID_ARGUMENT; msg = "s3 conf info err with role_arn or cred provider, please check it"; return; } + if (!obj->has_cred_provider_type()) { + obj->set_cred_provider_type(CredProviderTypePB::INSTANCE_PROFILE); + } } else { // ATTN: prefix may be empty if (plain_ak.empty() || plain_sk.empty() || bucket.empty() || endpoint.empty() || @@ -1071,7 +1075,11 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt msg = ss.str(); return -1; } - auto* storage_vault_names = instance.mutable_storage_vault_names(); + // Apply all mutations to a candidate. The caller can safely abandon the + // transaction on any validation/serialization error without leaking a + // partial rename into the InstanceInfoPB it later persists. + InstanceInfoPB candidate_instance = instance; + auto* storage_vault_names = candidate_instance.mutable_storage_vault_names(); auto* name_ptr = storage_vault_names->Mutable(pos); DCHECK(name_ptr != nullptr); const std::string old_name = *name_ptr; @@ -1081,8 +1089,8 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt target_vault_id, name, old_name); return -1; } - std::string vault_id = instance.resource_ids().Get(pos); - auto vault_key = storage_vault_key({instance.instance_id(), vault_id}); + std::string vault_id = candidate_instance.resource_ids().Get(pos); + auto vault_key = storage_vault_key({candidate_instance.instance_id(), vault_id}); std::string val; auto err = txn->get(vault_key, &val); @@ -1118,7 +1126,7 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt return -1; } - if (vault_exist(instance, vault.alter_name())) { + if (vault_exist(candidate_instance, vault.alter_name())) { code = MetaServiceCode::ALREADY_EXISTED; msg = fmt::format("vault_name={} already existed", vault.alter_name()); return -1; @@ -1126,14 +1134,15 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt new_vault.set_name(vault.alter_name()); *name_ptr = vault.alter_name(); - if (instance.default_storage_vault_id() == vault_id) { - instance.set_default_storage_vault_name(vault.alter_name()); + if (candidate_instance.default_storage_vault_id() == vault_id) { + candidate_instance.set_default_storage_vault_name(vault.alter_name()); } } - if (obj_info.has_role_arn() && (obj_info.has_ak() || obj_info.has_sk())) { + if ((obj_info.has_role_arn() || obj_info.has_cred_provider_type()) && + (obj_info.has_ak() || obj_info.has_sk())) { code = MetaServiceCode::INVALID_ARGUMENT; - msg = "invaild argument, both set ak/sk and role_arn is not allowed"; + msg = "invalid argument, access keys cannot be combined with role or credentials"; LOG(WARNING) << msg; return -1; } @@ -1168,14 +1177,41 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt new_vault.mutable_obj_info()->mutable_encryption_info()->CopyFrom(encryption_info); } - if (obj_info.has_role_arn()) { - new_vault.mutable_obj_info()->clear_ak(); - new_vault.mutable_obj_info()->clear_sk(); - new_vault.mutable_obj_info()->clear_encryption_info(); + if (obj_info.has_role_arn() || obj_info.has_cred_provider_type()) { + // Only selecting AWS authentication replaces other credentials. Clearing + // an unused role must preserve static keys. + if (!obj_info.role_arn().empty() || obj_info.has_cred_provider_type()) { + new_vault.mutable_obj_info()->clear_ak(); + new_vault.mutable_obj_info()->clear_sk(); + new_vault.mutable_obj_info()->clear_encryption_info(); + } + + if (obj_info.has_role_arn()) { + if (obj_info.role_arn().empty()) { + new_vault.mutable_obj_info()->clear_role_arn(); + // An external ID has no meaning without a role. Clearing the role + // explicitly clears its dependent value as well. + new_vault.mutable_obj_info()->clear_external_id(); + } else { + new_vault.mutable_obj_info()->set_role_arn(obj_info.role_arn()); + if (!new_vault.obj_info().has_cred_provider_type() && + !obj_info.has_cred_provider_type()) { + new_vault.mutable_obj_info()->set_cred_provider_type( + CredProviderTypePB::INSTANCE_PROFILE); + } + } + } + if (obj_info.has_cred_provider_type()) { + new_vault.mutable_obj_info()->set_cred_provider_type(obj_info.cred_provider_type()); + } + } - new_vault.mutable_obj_info()->set_role_arn(obj_info.role_arn()); - new_vault.mutable_obj_info()->set_cred_provider_type(get_cred_provider_type(obj_info)); - if (obj_info.has_external_id()) { + // external_id is an independent ALTER field. It may be changed without + // requiring FE to resend role_arn and cred_provider_type. + if (obj_info.has_external_id()) { + if (obj_info.external_id().empty()) { + new_vault.mutable_obj_info()->clear_external_id(); + } else { new_vault.mutable_obj_info()->set_external_id(obj_info.external_id()); } } @@ -1184,6 +1220,12 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt new_vault.mutable_obj_info()->set_use_path_style(obj_info.use_path_style()); } + if (auto error = validate_obj_authentication(new_vault.obj_info()); error.has_value()) { + code = MetaServiceCode::INVALID_ARGUMENT; + msg = *error; + return -1; + } + auto now_time = std::chrono::system_clock::now(); uint64_t time = std::chrono::duration_cast(now_time.time_since_epoch()).count(); @@ -1198,6 +1240,7 @@ static int alter_s3_storage_vault_by_id(InstanceInfoPB& instance, std::unique_pt } txn->put(vault_key, val); + instance.Swap(&candidate_instance); LOG(INFO) << "put vault_id=" << vault_id << ", vault_key=" << hex(vault_key) << ", origin vault=" << encryt_sk(hide_ak(origin_vault_info)) << ", new vault=" << encryt_sk(hide_ak(new_vault_info)); @@ -1620,7 +1663,9 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr break; } case AlterObjStoreInfoRequest::ALTER_S3_VAULT: { - alter_s3_storage_vault(instance, txn, request->vault(), code, msg, response); + if (alter_s3_storage_vault(instance, txn, request->vault(), code, msg, response) != 0) { + return; + } break; } case AlterObjStoreInfoRequest::ALTER_HDFS_VAULT: { diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index ec393bae1de7e5..edeffa841be261 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -35,6 +35,7 @@ #include #include #include +#include #include #include "common/config.h" @@ -752,6 +753,77 @@ TEST(MetaServiceTest, AlterS3StorageVaultTest) { i.ParseFromString(val); }; + auto get_test_vault = [&](std::string_view vault_id, StorageVaultPB& stored_vault) { + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({instance.instance_id(), vault_id}), &val), + TxnErrorCode::TXN_OK); + ASSERT_TRUE(stored_vault.ParseFromString(val)); + }; + + // Clearing an unused AWS role must preserve the active credentials, + // including encryption metadata, through the RPC and persistence boundary. + for (const auto* vault_id : {"2"}) { + StorageVaultPB stored; + get_test_vault(vault_id, stored); + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(stored.name()); + auto* update = req.mutable_vault()->mutable_obj_info(); + auto alter_vault = [&]() { + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, + nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + }; + if (stored.obj_info().has_ak()) { + update->set_ak("ak"); + update->set_sk("sk"); + alter_vault(); + get_test_vault(vault_id, stored); + ASSERT_TRUE(stored.obj_info().has_encryption_info()); + } + stored.mutable_obj_info()->clear_mtime(); + const auto original = stored.obj_info().SerializeAsString(); + update->Clear(); + update->set_role_arn(""); + // Retrying the same field clear must also preserve the credentials. + for (int attempt = 0; attempt < 2; ++attempt) { + alter_vault(); + get_test_vault(vault_id, stored); + stored.mutable_obj_info()->clear_mtime(); + ASSERT_EQ(stored.obj_info().SerializeAsString(), original); + } + } + + // A rejected credential update must not commit a rename that was applied + // earlier in the request. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(vault_name); + req.mutable_vault()->set_alter_name("rename_must_not_commit"); + req.mutable_vault()->mutable_obj_info()->set_ak("unpaired-key"); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT) << res.status().msg(); + + InstanceInfoPB stored_instance; + get_test_instance(stored_instance); + ASSERT_EQ(stored_instance.storage_vault_names(0), vault_name); + StorageVaultPB stored_vault; + get_test_vault("2", stored_vault); + ASSERT_EQ(stored_vault.name(), vault_name); + } + { AlterObjStoreInfoRequest req; req.set_cloud_unique_id("test_cloud_unique_id"); @@ -4948,6 +5020,48 @@ TEST(MetaServiceTest, CalcSyncVersionsTest) { } } +TEST(MetaServiceTest, AddRoleOnlyVaultDefaultsToInstanceProfile) { + auto meta_service = get_meta_service(); + brpc::Controller cntl; + std::string key; + instance_key({"test_instance"}, &key); + InstanceInfoPB instance; + instance.set_instance_id("test_instance"); + instance.set_enable_storage_vault(true); + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + txn->put(key, instance.SerializeAsString()); + ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK); + + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ADD_S3_VAULT); + req.mutable_vault()->set_name("role_only_vault"); + auto* obj = req.mutable_obj(); + obj->set_provider(ObjectStoreInfoPB::S3); + obj->set_bucket("bucket"); + obj->set_endpoint("s3.us-east-1.amazonaws.com"); + obj->set_region("us-east-1"); + obj->set_role_arn("arn:aws:iam::123456789012:role/test"); + ASSERT_FALSE(obj->has_cred_provider_type()); + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault(&cntl, &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + txn.reset(); + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(key, &val), TxnErrorCode::TXN_OK); + ASSERT_TRUE(instance.ParseFromString(val)); + ASSERT_EQ(instance.resource_ids_size(), 1); + ASSERT_EQ(txn->get(storage_vault_key({"test_instance", instance.resource_ids(0)}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB vault; + ASSERT_TRUE(vault.ParseFromString(val)); + EXPECT_EQ(vault.obj_info().role_arn(), obj->role_arn()); + EXPECT_EQ(vault.obj_info().cred_provider_type(), CredProviderTypePB::INSTANCE_PROFILE); +} + TEST(MetaServiceTest, StageTest) { auto meta_service = get_meta_service(); brpc::Controller cntl; @@ -13299,6 +13413,87 @@ TEST(MetaServiceTest, AlterS3StorageVaultWithRoleArnTest) { ASSERT_EQ(get_obj.name(), new_vault_name) << get_obj.obj_info().ShortDebugString(); } + // Role, external ID, and credential provider are independent ALTER fields. + // Updating one must preserve the other two. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name("new_test_alter_s3_vault_111"); + req.mutable_vault()->mutable_obj_info()->set_cred_provider_type( + CredProviderTypePB::CONTAINER); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({"GetObjStoreInfoTestInstance", "2"}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB stored_vault; + ASSERT_TRUE(stored_vault.ParseFromString(val)); + ASSERT_EQ(stored_vault.obj_info().role_arn(), "arn:aws:iam::12311321:role/test-alter-role"); + ASSERT_EQ(stored_vault.obj_info().external_id(), "external_id_123123"); + ASSERT_EQ(stored_vault.obj_info().cred_provider_type(), CredProviderTypePB::CONTAINER); + } + + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name("new_test_alter_s3_vault_111"); + req.mutable_vault()->mutable_obj_info()->set_role_arn( + "arn:aws:iam::12311321:role/replacement-role"); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({"GetObjStoreInfoTestInstance", "2"}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB stored_vault; + ASSERT_TRUE(stored_vault.ParseFromString(val)); + ASSERT_EQ(stored_vault.obj_info().role_arn(), + "arn:aws:iam::12311321:role/replacement-role"); + ASSERT_EQ(stored_vault.obj_info().external_id(), "external_id_123123"); + ASSERT_EQ(stored_vault.obj_info().cred_provider_type(), CredProviderTypePB::CONTAINER); + } + + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name("new_test_alter_s3_vault_111"); + req.mutable_vault()->mutable_obj_info()->set_external_id("replacement-external-id"); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::OK) << res.status().msg(); + + std::unique_ptr txn; + ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK); + std::string val; + ASSERT_EQ(txn->get(storage_vault_key({"GetObjStoreInfoTestInstance", "2"}), &val), + TxnErrorCode::TXN_OK); + StorageVaultPB stored_vault; + ASSERT_TRUE(stored_vault.ParseFromString(val)); + ASSERT_EQ(stored_vault.obj_info().role_arn(), + "arn:aws:iam::12311321:role/replacement-role"); + ASSERT_EQ(stored_vault.obj_info().external_id(), "replacement-external-id"); + ASSERT_EQ(stored_vault.obj_info().cred_provider_type(), CredProviderTypePB::CONTAINER); + } + std::string cipher_sk = "JUkuTDctR+ckJtnPkLScWaQZRcOtWBhsLLpnCRxQLxr734qB8cs6gNLH6grE1FxO"; std::string plain_sk = "Hx60p12123af234541nsVsffdfsdfghsdfhsdf34t"; diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java index c5851405958e0d..62167919196afc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java @@ -38,6 +38,7 @@ import java.io.IOException; import java.util.Arrays; import java.util.HashSet; +import java.util.Locale; import java.util.Map; import java.util.Set; import java.util.stream.Collectors; @@ -103,6 +104,10 @@ public void modifyProperties(ImmutableMap newProperties) throws for (Map.Entry kv : newProperties.entrySet()) { replaceIfEffectiveValue(this.properties, kv.getKey(), kv.getValue()); } + // Reject object-storage properties before they can select a different filesystem + // during the connectivity check, even when that check is disabled. + // Leave Kerberos validation in filesystem binding to preserve its error message. + this.properties.keySet().forEach(HdfsStorageVault::checkForbiddenProperty); checkConnectivity(this.properties); } @@ -192,23 +197,7 @@ public static Cloud.HdfsVaultInfo generateHdfsParam(Map properti } else if (property.getKey().equalsIgnoreCase(StorageVault.PropertyKey.VAULT_NAME)) { continue; } else { - // Get rid of copy and paste from create s3 vault stmt - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.REGION), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.ENDPOINT), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.ROOT_PATH), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.PROVIDER), - "Invalid argument %s", property.getKey()); - Preconditions.checkArgument( - !property.getKey().toLowerCase().contains(S3StorageVault.PropertyKey.BUCKET), - "Invalid argument %s", property.getKey()); - + checkForbiddenProperty(property.getKey()); if (!NON_HDFS_CONF_PROPERTY_KEYS.contains(property.getKey().toLowerCase())) { Cloud.HdfsBuildConf.HdfsConfKVPair.Builder conf = Cloud.HdfsBuildConf.HdfsConfKVPair.newBuilder(); conf.setKey(property.getKey()); @@ -219,4 +208,19 @@ public static Cloud.HdfsVaultInfo generateHdfsParam(Map properti } return hdfsVaultInfoBuilder.setBuildConf(hdfsConfBuilder.build()).build(); } + + private static void checkForbiddenProperty(String key) { + // Get rid of copy and paste from create s3 vault stmt. + String lowerCaseKey = key.toLowerCase(Locale.ROOT); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.REGION), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.ENDPOINT), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.ROOT_PATH), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.PROVIDER), + "Invalid argument %s", key); + Preconditions.checkArgument(!lowerCaseKey.contains(S3StorageVault.PropertyKey.BUCKET), + "Invalid argument %s", key); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java index 0c70cd67ce4a26..c7d798d5dbdfac 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/S3Resource.java @@ -45,6 +45,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.Optional; /** @@ -219,71 +220,79 @@ protected static void pingS3(String bucketName, String rootPath, Map properties) throws DdlException { + public synchronized void modifyProperties(Map newProperties) throws DdlException { + // Serialize the snapshot, validation and publication. A lock only around publication + // would allow a concurrent ALTER to replace a successful update with an older snapshot. + Map properties = new HashMap<>(newProperties); + // Normalize the patch independently so legacy aliases override persisted values. + S3ResourceCompat.convertToStdProperties(properties); + Map normalizedUpdates = properties; + Map effectiveProperties = new HashMap<>(this.properties); + S3ResourceCompat.convertToStdProperties(effectiveProperties); + for (Map.Entry update : normalizedUpdates.entrySet()) { + // Empty updates are ignored, except when clearing a session token. + replaceIfEffectiveValue(effectiveProperties, update.getKey(), update.getValue()); + if (S3ResourceCompat.SESSION_TOKEN.equals(update.getKey()) + || S3ResourceCompat.Env.TOKEN.equals(update.getKey())) { + effectiveProperties.put(update.getKey(), update.getValue()); + } + } if (references.containsValue(ReferenceType.POLICY)) { // can't change, because remote fs use it info to find data. List cantChangeProperties = Arrays.asList(S3ResourceCompat.ENDPOINT, S3ResourceCompat.REGION, S3ResourceCompat.ROOT_PATH, S3ResourceCompat.BUCKET, S3ResourceCompat.Env.ENDPOINT, S3ResourceCompat.Env.REGION, S3ResourceCompat.Env.ROOT_PATH, S3ResourceCompat.Env.BUCKET); - Optional any = cantChangeProperties.stream().filter(properties::containsKey).findAny(); + Optional any = cantChangeProperties.stream() + .filter(key -> normalizedUpdates.containsKey(key) + || !Objects.equals(this.properties.get(key), effectiveProperties.get(key))) + .findAny(); if (any.isPresent()) { throw new DdlException("current not support modify property : " + any.get()); } } - // compatible with old version, Need convert if modified properties map uses old properties. - S3ResourceCompat.convertToStdProperties(properties); - if (!Strings.isNullOrEmpty(properties.get(S3ResourceCompat.ENDPOINT))) { - properties.put(S3ResourceCompat.Env.ENDPOINT, properties.get(S3ResourceCompat.ENDPOINT)); + if (!Strings.isNullOrEmpty(effectiveProperties.get(S3ResourceCompat.ENDPOINT))) { + effectiveProperties.put(S3ResourceCompat.Env.ENDPOINT, effectiveProperties.get(S3ResourceCompat.ENDPOINT)); } - boolean needCheck = isNeedCheck(properties); + for (Map.Entry kv : normalizedUpdates.entrySet()) { + if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ROLE_ARN) + && !Strings.isNullOrEmpty(kv.getValue())) { + effectiveProperties.remove(S3ResourceCompat.ACCESS_KEY); + effectiveProperties.remove(S3ResourceCompat.Env.ACCESS_KEY); + effectiveProperties.remove(S3ResourceCompat.SECRET_KEY); + effectiveProperties.remove(S3ResourceCompat.Env.SECRET_KEY); + } + if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ACCESS_KEY) + && !Strings.isNullOrEmpty(kv.getValue())) { + effectiveProperties.remove(S3ResourceCompat.ROLE_ARN); + effectiveProperties.remove(S3ResourceCompat.Env.ROLE_ARN); + effectiveProperties.remove(S3ResourceCompat.EXTERNAL_ID); + effectiveProperties.remove(S3ResourceCompat.Env.EXTERNAL_ID); + } + } + boolean needCheck = isNeedCheck(effectiveProperties); if (LOG.isDebugEnabled()) { LOG.debug("s3 info need check validity : {}", needCheck); } if (needCheck) { - S3ResourceCompat.requiredS3PingProperties(this.properties); - Map changedProperties = new HashMap<>(this.properties); - changedProperties.putAll(properties); + S3ResourceCompat.requiredS3PingProperties(effectiveProperties); + Map changedProperties = new HashMap<>(effectiveProperties); String endpoint = S3Util.buildEndpointUrl(changedProperties.get(S3ResourceCompat.ENDPOINT)); changedProperties.put(S3ResourceCompat.ENDPOINT, endpoint); changedProperties.put(S3ResourceCompat.Env.ENDPOINT, endpoint); - String bucketName = properties.getOrDefault(S3ResourceCompat.BUCKET, - this.properties.get(S3ResourceCompat.BUCKET)); - String rootPath = properties.getOrDefault(S3ResourceCompat.ROOT_PATH, - this.properties.get(S3ResourceCompat.ROOT_PATH)); - - pingS3(bucketName, rootPath, changedProperties); + pingS3(effectiveProperties.get(S3ResourceCompat.BUCKET), + effectiveProperties.get(S3ResourceCompat.ROOT_PATH), + changedProperties); } - // modify properties writeLock(); - - for (Map.Entry kv : properties.entrySet()) { - replaceIfEffectiveValue(this.properties, kv.getKey(), kv.getValue()); - if (kv.getKey().equals(S3ResourceCompat.Env.TOKEN) - || kv.getKey().equals(S3ResourceCompat.SESSION_TOKEN)) { - this.properties.put(kv.getKey(), kv.getValue()); - } - - if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ROLE_ARN) - && !Strings.isNullOrEmpty(kv.getValue())) { - this.properties.remove(S3ResourceCompat.ACCESS_KEY); - this.properties.remove(S3ResourceCompat.Env.ACCESS_KEY); - this.properties.remove(S3ResourceCompat.SECRET_KEY); - this.properties.remove(S3ResourceCompat.Env.SECRET_KEY); - } - - if (kv.getKey().equalsIgnoreCase(S3ResourceCompat.ACCESS_KEY) - && !Strings.isNullOrEmpty(kv.getValue())) { - this.properties.remove(S3ResourceCompat.ROLE_ARN); - this.properties.remove(S3ResourceCompat.Env.ROLE_ARN); - this.properties.remove(S3ResourceCompat.EXTERNAL_ID); - this.properties.remove(S3ResourceCompat.Env.EXTERNAL_ID); - } + try { + this.properties = effectiveProperties; + ++version; + } finally { + writeUnlock(); } - ++version; - writeUnlock(); - super.modifyProperties(properties); + super.modifyProperties(effectiveProperties); } private CloudCredentialWithEndpoint getS3PingCredentials(Map properties) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java index 4662c72d7ff621..fd6976ecbe5e46 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/StorageVaultMgr.java @@ -26,6 +26,7 @@ import org.apache.doris.common.UserException; import org.apache.doris.common.lock.MonitoredReentrantReadWriteLock; import org.apache.doris.datasource.storage.CloudObjectStoreAdapter; +import org.apache.doris.datasource.storage.S3ResourceCompat; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.proto.InternalService.PAlterVaultSyncRequest; import org.apache.doris.rpc.BackendServiceProxy; @@ -145,16 +146,36 @@ private void updateDefaultStorageVaultCache(Pair newDefaultVault } } - private Cloud.StorageVaultPB.Builder buildAlterS3VaultRequest(Map properties, String name) + private Cloud.StorageVaultPB.Builder buildS3VaultRequest(Map properties, String name) throws Exception { - Cloud.ObjectStoreInfoPB.Builder objBuilder = CloudObjectStoreAdapter.getObjStoreInfoPB(properties); - Cloud.StorageVaultPB.Builder alterObjVaultBuilder = Cloud.StorageVaultPB.newBuilder(); - alterObjVaultBuilder.setName(name); - alterObjVaultBuilder.setObjInfo(objBuilder.build()); + Cloud.StorageVaultPB.Builder builder = Cloud.StorageVaultPB.newBuilder() + .setName(name).setObjInfo(CloudObjectStoreAdapter.getObjStoreInfoPB(properties)); if (properties.containsKey(StorageVault.PropertyKey.VAULT_NAME)) { - alterObjVaultBuilder.setAlterName(properties.get(StorageVault.PropertyKey.VAULT_NAME)); + builder.setAlterName(properties.get(StorageVault.PropertyKey.VAULT_NAME)); + } + return builder; + } + + private Cloud.StorageVaultPB.Builder buildAlterS3VaultRequest(Map properties, String name) + throws Exception { + Cloud.StorageVaultPB.Builder builder = buildS3VaultRequest(properties, name); + Cloud.ObjectStoreInfoPB.Builder objBuilder = builder.getObjInfoBuilder(); + // getObjStoreInfoPB supplies INSTANCE_PROFILE as the create-time default + // when role_arn is present. ALTER is a patch operation, so only send the + // credential provider when the user explicitly changed it. + if (!properties.containsKey(S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE) + && !properties.containsKey(S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE)) { + objBuilder.clearCredProviderType(); } - return alterObjVaultBuilder; + // Empty strings are meaningful for ALTER: they explicitly clear the + // stored role or external ID. The generic create builder omits them. + if (properties.containsKey(S3ResourceCompat.ROLE_ARN)) { + objBuilder.setRoleArn(properties.get(S3ResourceCompat.ROLE_ARN)); + } + if (properties.containsKey(S3ResourceCompat.EXTERNAL_ID)) { + objBuilder.setExternalId(properties.get(S3ResourceCompat.EXTERNAL_ID)); + } + return builder; } private Cloud.StorageVaultPB.Builder buildAlterHdfsVaultRequest(Map properties, String name) @@ -182,9 +203,15 @@ private Cloud.StorageVaultPB.Builder buildAlterStorageVaultRequest(StorageVaultT return builder; } - private Cloud.StorageVaultPB.Builder buildAlterStorageVaultRequest(StorageVault vault) throws Exception { - Cloud.StorageVaultPB.Builder builder = buildAlterStorageVaultRequest(vault.getType(), - vault.getCopiedProperties(), vault.getName()); + private Cloud.StorageVaultPB.Builder buildCreateStorageVaultRequest(StorageVault vault) throws Exception { + Cloud.StorageVaultPB.Builder builder; + if (vault.getType() == StorageVaultType.S3) { + builder = buildS3VaultRequest(vault.getCopiedProperties(), vault.getName()); + } else if (vault.getType() == StorageVaultType.HDFS) { + builder = buildAlterHdfsVaultRequest(vault.getCopiedProperties(), vault.getName()); + } else { + throw new DdlException("Unknown storage vault type"); + } Cloud.StorageVaultPB.PathFormat.Builder pathBuilder = Cloud.StorageVaultPB.PathFormat.newBuilder(); pathBuilder.setShardNum(vault.getNumShard()); pathBuilder.setPathVersion(vault.getPathVersion()); @@ -209,8 +236,8 @@ public void alterStorageVault(StorageVaultType type, Map propert request.setOp(Operation.ALTER_S3_VAULT); } else if (type == StorageVaultType.HDFS) { properties.keySet().stream() - // Exact literals of the legacy S3Properties.S3_PREFIX ("s3.") and - // StorageProperties.FS_PROVIDER_KEY ("provider") constants. + // Exact literals of the legacy S3ResourceCompat.S3_PREFIX ("s3.") and + // "provider" ("provider") constants. .filter(key -> HdfsStorageVault.FORBID_ALTER_PROPERTIES.contains(key) || key.toLowerCase().contains("s3.") || key.toLowerCase().contains("provider")) @@ -321,7 +348,7 @@ public StorageVaultType getStorageVaultTypeByName(String vaultName) throws DdlEx @VisibleForTesting public void createHdfsVault(StorageVault vault) throws Exception { - Cloud.StorageVaultPB.Builder alterHdfsInfoBuilder = buildAlterStorageVaultRequest(vault); + Cloud.StorageVaultPB.Builder alterHdfsInfoBuilder = buildCreateStorageVaultRequest(vault); Cloud.AlterObjStoreInfoRequest.Builder requestBuilder = Cloud.AlterObjStoreInfoRequest.newBuilder().setRequestIp(FrontendOptions.getLocalHostAddressCached()); requestBuilder.setOp(Cloud.AlterObjStoreInfoRequest.Operation.ADD_HDFS_INFO); @@ -371,7 +398,7 @@ private void alterSyncVaultTask() { } public void createS3Vault(StorageVault vault) throws Exception { - Cloud.StorageVaultPB.Builder s3StorageVaultBuilder = buildAlterStorageVaultRequest(vault); + Cloud.StorageVaultPB.Builder s3StorageVaultBuilder = buildCreateStorageVaultRequest(vault); Cloud.AlterObjStoreInfoRequest.Builder requestBuilder = Cloud.AlterObjStoreInfoRequest.newBuilder().setRequestIp(FrontendOptions.getLocalHostAddressCached()); requestBuilder.setOp(Cloud.AlterObjStoreInfoRequest.Operation.ADD_S3_VAULT); diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java index 57dff9d4d38ed9..a7c97e7f4dfc43 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/S3ResourceTest.java @@ -48,6 +48,10 @@ import java.nio.file.Paths; import java.util.HashMap; import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; public class S3ResourceTest { private static final Logger LOG = LogManager.getLogger(S3ResourceTest.class); @@ -213,6 +217,69 @@ public void testSerialization() throws Exception { Files.deleteIfExists(path); } + @Test + public void testConcurrentAltersPreserveBothUpdates() throws Exception { + CountDownLatch firstReady = new CountDownLatch(1); + CountDownLatch releaseFirst = new CountDownLatch(1); + CountDownLatch secondStarted = new CountDownLatch(1); + AtomicBoolean first = new AtomicBoolean(true); + AtomicReference failure = new AtomicReference<>(); + S3Resource resource = new S3Resource("concurrent_alter") { + @Override + public void writeLock() { + if (first.compareAndSet(true, false)) { + firstReady.countDown(); + try { + Assertions.assertTrue(releaseFirst.await(10, TimeUnit.SECONDS)); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } + super.writeLock(); + } + }; + resource.setProperties(ImmutableMap.copyOf(s3Properties)); + Thread updateConnections = new Thread(() -> { + try { + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.MAX_CONNECTIONS, "77")); + } catch (Throwable e) { + failure.compareAndSet(null, e); + } + }); + Thread updateTimeout = new Thread(() -> { + secondStarted.countDown(); + try { + resource.modifyProperties(ImmutableMap.of(S3ResourceCompat.REQUEST_TIMEOUT_MS, "8888")); + } catch (Throwable e) { + failure.compareAndSet(null, e); + } + }); + try { + updateConnections.start(); + Assertions.assertTrue(firstReady.await(10, TimeUnit.SECONDS)); + updateTimeout.start(); + Assertions.assertTrue(secondStarted.await(10, TimeUnit.SECONDS)); + // The second ALTER either waits for the first one's monitor, or (before the fix) + // finishes and is subsequently overwritten by the first one's stale snapshot. + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (updateTimeout.isAlive() && updateTimeout.getState() != Thread.State.BLOCKED + && System.nanoTime() < deadline) { + Thread.yield(); + } + Assertions.assertTrue(!updateTimeout.isAlive() || updateTimeout.getState() == Thread.State.BLOCKED); + } finally { + releaseFirst.countDown(); + updateConnections.join(10000); + updateTimeout.join(10000); + } + Assertions.assertFalse(updateConnections.isAlive()); + Assertions.assertFalse(updateTimeout.isAlive()); + Assertions.assertNull(failure.get()); + Assertions.assertEquals("77", resource.getProperty(S3ResourceCompat.MAX_CONNECTIONS)); + Assertions.assertEquals("8888", resource.getProperty(S3ResourceCompat.REQUEST_TIMEOUT_MS)); + } + @Test public void testModifyProperties() throws Exception { ImmutableMap properties = ImmutableMap.of( diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java index 2657aede6ad857..03d292b7442d4d 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/HdfsStorageVaultTest.java @@ -27,6 +27,7 @@ import org.apache.doris.cloud.rpc.MetaServiceProxy; import org.apache.doris.common.Config; import org.apache.doris.common.DdlException; +import org.apache.doris.datasource.storage.StorageAdapter; import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; import org.apache.doris.system.SystemInfoService; @@ -42,6 +43,8 @@ import org.mockito.Mockito; import java.util.HashSet; +import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Set; @@ -262,6 +265,48 @@ public void testCheckConnectivityException() { }); } + @Test + public void testRejectS3PropertiesBeforeConnectivity() { + try (MockedStatic adapter = Mockito.mockStatic(StorageAdapter.class)) { + for (String key : new String[] {"s3.region", "s3.endpoint", "s3.root.path", "provider", "s3.bucket"}) { + for (String invalidKey : new String[] {key, key.toUpperCase(Locale.ROOT)}) { + for (String validityCheck : new String[] {"true", "false"}) { + ImmutableMap properties = ImmutableMap.of( + "FS.DEFAULTFS", "hdfs://localhost:10000", + HdfsStorageVault.PropertyKey.VAULT_PATH_PREFIX, "testInvalidProperties", + HdfsStorageVault.S3_VALIDITY_CHECK, validityCheck, + invalidKey, "invalid"); + HdfsStorageVault vault = new HdfsStorageVault("testHdfsVault", false, false); + IllegalArgumentException exception = Assertions.assertThrows( + IllegalArgumentException.class, () -> vault.modifyProperties(properties)); + Assertions.assertEquals("Invalid argument " + invalidKey, exception.getMessage()); + } + } + } + adapter.verifyNoInteractions(); + } + } + + @Test + public void testKerberosValidationBeforeConnectivity() { + for (Map credentials : List.>of( + ImmutableMap.of(), + ImmutableMap.of(HdfsStorageVault.PropertyKey.HADOOP_KERBEROS_PRINCIPAL, "hadoop/localhost@EXAMPLE.COM"), + ImmutableMap.of(HdfsStorageVault.PropertyKey.HADOOP_KERBEROS_KEYTAB, "/path/to/test.keytab"))) { + ImmutableMap properties = ImmutableMap.builder() + .put(HdfsStorageVault.PropertyKey.HADOOP_FS_NAME, "hdfs://localhost:10000") + .put(HdfsStorageVault.PropertyKey.VAULT_PATH_PREFIX, "testKerberosProperties") + .put(HdfsStorageVault.PropertyKey.HADOOP_SECURITY_AUTHENTICATION, "kerberos") + .putAll(credentials) + .build(); + HdfsStorageVault vault = new HdfsStorageVault("testHdfsVault", false, false); + IllegalArgumentException exception = Assertions.assertThrows( + IllegalArgumentException.class, () -> vault.modifyProperties(properties)); + Assertions.assertEquals("HDFS authentication type is kerberos, but principal or keytab is not set.", + exception.getMessage()); + } + } + @Test public void testIgnoreCheckConnectivity() throws DdlException { ImmutableMap properties = ImmutableMap.of( diff --git a/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java new file mode 100644 index 00000000000000..e0480eac8483df --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/S3StorageVaultTest.java @@ -0,0 +1,126 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.cloud.catalog; + +import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.StorageVault; +import org.apache.doris.catalog.StorageVault.StorageVaultType; +import org.apache.doris.catalog.StorageVaultMgr; +import org.apache.doris.cloud.proto.Cloud; +import org.apache.doris.cloud.rpc.MetaServiceProxy; +import org.apache.doris.common.DdlException; +import org.apache.doris.datasource.storage.S3ResourceCompat; +import org.apache.doris.nereids.trees.plans.commands.CreateStorageVaultCommand; +import org.apache.doris.system.SystemInfoService; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.atomic.AtomicReference; + +public class S3StorageVaultTest { + private static final String ROLE = "arn:aws:iam::123456789012:role/test-role"; + + // Capture the actual CREATE/ALTER RPC, then stop before cache updates and backend synchronization. + private Cloud.ObjectStoreInfoPB captureRequest(Map auth, boolean isAlter) throws Exception { + AtomicReference captured = new AtomicReference<>(); + Env env = Mockito.mock(Env.class); + MetaServiceProxy proxy = Mockito.mock(MetaServiceProxy.class); + try (MockedStatic mockedProxy = Mockito.mockStatic(MetaServiceProxy.class); + MockedStatic mockedEnv = Mockito.mockStatic(Env.class)) { + mockedProxy.when(MetaServiceProxy::getInstance).thenReturn(proxy); + mockedEnv.when(Env::getCurrentEnv).thenReturn(env); + Mockito.when(proxy.alterStorageVault(Mockito.any(Cloud.AlterObjStoreInfoRequest.class))) + .thenAnswer(invocation -> { + captured.set(invocation.getArgument(0)); + return Cloud.AlterObjStoreInfoResponse.newBuilder().setStatus( + Cloud.MetaServiceResponseStatus.newBuilder() + .setCode(Cloud.MetaServiceCode.INVALID_ARGUMENT) + .setMsg("captured vault request")).build(); + }); + StorageVaultMgr mgr = new StorageVaultMgr(new SystemInfoService()); + if (isAlter) { + Assertions.assertThrows(DdlException.class, + () -> mgr.alterStorageVault(StorageVaultType.S3, new HashMap<>(auth), "iam_vault")); + } else { + Map properties = new HashMap<>(auth); + properties.put("type", "S3"); + properties.put("provider", "S3"); + properties.put(S3ResourceCompat.ENDPOINT, "s3.us-east-1.amazonaws.com"); + properties.put(S3ResourceCompat.REGION, "us-east-1"); + properties.put(S3ResourceCompat.BUCKET, "test-bucket"); + properties.put(S3ResourceCompat.ROOT_PATH, "test-root"); + properties.put(S3ResourceCompat.VALIDITY_CHECK, "false"); + CreateStorageVaultCommand command = new CreateStorageVaultCommand(false, "iam_vault", properties); + command.setStorageVaultType(StorageVaultType.S3); + StorageVault vault = StorageVault.fromCommand(command); + Assertions.assertThrows(DdlException.class, () -> mgr.createS3Vault(vault)); + } + Assertions.assertNotNull(captured.get(), "The request must reach the MS RPC boundary"); + Assertions.assertEquals(isAlter ? Cloud.AlterObjStoreInfoRequest.Operation.ALTER_S3_VAULT + : Cloud.AlterObjStoreInfoRequest.Operation.ADD_S3_VAULT, captured.get().getOp()); + if (isAlter) { + Assertions.assertFalse(captured.get().getVault().getObjInfo().hasProvider(), + "ALTER must not send an inferred immutable provider"); + } + return captured.get().getVault().getObjInfo(); + } + } + + @Test + public void testAwsCreatePreservesDefaultAndExplicitProvider() throws Exception { + Map properties = new HashMap<>(); + properties.put(S3ResourceCompat.ROLE_ARN, ROLE); + Cloud.ObjectStoreInfoPB obj = captureRequest(properties, false); + Assertions.assertTrue(obj.hasCredProviderType()); + Assertions.assertEquals(Cloud.CredProviderTypePB.INSTANCE_PROFILE, obj.getCredProviderType()); + for (String key : new String[] {S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, + S3ResourceCompat.Env.CREDENTIALS_PROVIDER_TYPE}) { + Map explicit = new HashMap<>(properties); + explicit.put(key, "CONTAINER"); + obj = captureRequest(explicit, false); + Assertions.assertTrue(obj.hasCredProviderType()); + Assertions.assertEquals(Cloud.CredProviderTypePB.CONTAINER, obj.getCredProviderType()); + } + } + + @Test + public void testAwsAlterPreservesOmissionAndExplicitClears() throws Exception { + Map properties = new HashMap<>(); + properties.put(S3ResourceCompat.ROLE_ARN, ROLE); + Assertions.assertFalse(captureRequest(properties, true).hasCredProviderType()); + properties.put(S3ResourceCompat.CREDENTIALS_PROVIDER_TYPE, "DEFAULT"); + Cloud.ObjectStoreInfoPB obj = captureRequest(properties, true); + Assertions.assertTrue(obj.hasCredProviderType()); + Assertions.assertEquals(Cloud.CredProviderTypePB.DEFAULT, obj.getCredProviderType()); + properties.clear(); + properties.put(S3ResourceCompat.ROLE_ARN, ""); + properties.put(S3ResourceCompat.EXTERNAL_ID, ""); + obj = captureRequest(properties, true); + Assertions.assertTrue(obj.hasRoleArn()); + Assertions.assertEquals("", obj.getRoleArn()); + Assertions.assertTrue(obj.hasExternalId()); + Assertions.assertEquals("", obj.getExternalId()); + Assertions.assertFalse(obj.hasCredProviderType()); + } + +} From 9f13b4310b506e03e90517a4be333695411ffa32 Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 23 Oct 2026 21:10:11 +0800 Subject: [PATCH 46/47] [refactor](fe) Separate filesystem routing validation and fallback hooks ### What problem does this PR solve? Related PR: #68540 Problem Summary: Filesystem providers that overlap with a generic provider need an explicit fallback relationship, and provider-owned invalid properties must not disappear when another provider is selected. Add two default SPI hooks, apply them to primary/all/direct binding, and retain explicit filesystem flags and existing ordering. This prerequisite contains no GCP authentication or protocol changes. Bump the filesystem plugin API from 1.0 to 2.0. ### Release note Filesystem plugins can validate owned properties before routing and declare heuristic fallback providers. Rebuild plugins against filesystem API 2.0. ### Check List (For Author) - Test: FE Core/SPI Checkstyle, Java parsing, and git diff --check passed. Added provider-order, explicit-selection, fallback and validation tests. Standard FE UT could not reach test execution because the configured protoc executable was unavailable; unit tests were not executed. - Behavior changed: Existing providers retain their default behavior; providers can opt into validation and fallback hooks. - Does this need documentation: Yes, filesystem README updated. --- .../datasource/storage/StorageAdapter.java | 3 + .../doris/fs/FileSystemPluginManager.java | 61 +++++-- .../doris/fs/FileSystemPluginManagerTest.java | 151 ++++++++++++++++++ fe/fe-filesystem/README.md | 17 ++ .../filesystem/spi/FileSystemProvider.java | 17 ++ .../resources/filesystem-plugin-surface.txt | 2 + fe/fe-filesystem/pom.xml | 2 +- 7 files changed, 242 insertions(+), 11 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java index 0c42c095d33cff..a8bec03bf67a7f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java @@ -170,6 +170,9 @@ public static List ofAll(Map origProps) { */ public static StorageAdapter ofProvider(String providerName, Map origProps) { Map props = origProps == null ? new HashMap<>() : origProps; + Map selection = new HashMap<>(props); + selection.put("provider", providerName); + manager().validateProperties(selection); for (FileSystemProvider provider : manager().getProviders()) { if (provider.name().equalsIgnoreCase(providerName)) { return new StorageAdapter(provider.bind(withHadoopConfigDir(props)), props); diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java index 4c3d59548307cf..4e72cd2427aa6b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemPluginManager.java @@ -45,6 +45,7 @@ import java.util.Map; import java.util.ServiceLoader; import java.util.Set; +import java.util.TreeSet; import java.util.concurrent.CopyOnWriteArrayList; /** @@ -54,8 +55,8 @@ * 1. ServiceLoader scan (classpath-based built-ins / test overrides) * 2. DirectoryPluginRuntimeManager scan (production plugin directories) * - *

The first provider that returns {@code supports(props) == true} is used. - * Classpath providers have higher priority than directory-loaded providers. + *

The first supporting provider is used unless a matching provider supersedes its declared + * heuristic fallbacks. Classpath providers have higher priority than directory-loaded providers. */ public class FileSystemPluginManager { @@ -200,8 +201,16 @@ private boolean hasProviderNamed(String name) { * @throws IOException if no provider supports the properties, or creation fails */ public FileSystem createFileSystem(Map properties) throws IOException { + validateProperties(properties); + Set fallbacks = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (FileSystemProvider provider : providers) { + if (!provider.fallbackProviders().isEmpty() && provider.supports(properties)) { + fallbacks.addAll(provider.fallbackProviders()); + } + } for (FileSystemProvider provider : providers) { - if (provider.supports(properties)) { + if ((!fallbacks.contains(provider.name()) || provider.supportsExplicit(properties)) + && provider.supports(properties)) { return provider.create(properties); } } @@ -228,32 +237,61 @@ public List getProviders() { private static final String DEPRECATED_OSS_HDFS_SUPPORT = "oss.hdfs.enabled"; + /** Validate plugin-owned keys even if routing would otherwise ignore them. */ + public void validateProperties(Map properties) { + for (FileSystemProvider provider : providers) { + provider.validateProperties(properties); + } + } + + private Set fallbackProviders(Map properties, Map probeView, + boolean useGuess) { + Set fallbacks = new TreeSet<>(String.CASE_INSENSITIVE_ORDER); + for (FileSystemProvider provider : providers) { + if (!provider.fallbackProviders().isEmpty() + && (provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView)))) { + fallbacks.addAll(provider.fallbackProviders()); + } + } + return fallbacks; + } + + private static boolean matches(FileSystemProvider provider, Map properties, + Map probeView, boolean useGuess, Set fallbacks) { + return provider.supportsExplicit(properties) + || (useGuess && !fallbacks.contains(provider.name()) && provider.supportsGuess(probeView)); + } + /** * Binds the first matching provider for raw user properties, mirroring * {@code StorageProperties.createPrimary}: fixed priority, explicit {@code fs..support} - * flags first-class, heuristic guesses globally disabled once any explicit flag is present, + * flags first-class, heuristic guesses disabled by an explicit flag, + * matched plugins superseding their declared heuristic fallbacks, * and no default fallback — throws when nothing matches. */ public FileSystemProperties bindPrimary(Map properties) { + validateProperties(properties); boolean useGuess = !hasAnyExplicitFsSupport(properties); Map probeView = withProbeContext(properties); + Set fallbacks = fallbackProviders(properties, probeView, useGuess); for (StorageRegistry.Provider meta : StorageRegistry.Provider.values()) { FileSystemProvider provider = providerByName(meta.name()); if (provider == null) { continue; } - if (provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView))) { + if (matches(provider, properties, probeView, useGuess, fallbacks)) { return provider.bind(properties); } } // Providers not in the StorageRegistry.Provider table (out-of-tree plugins) are consulted // after every known provider, in registration order — adding a plugin needs no fe-core - // priority-list change and can never preempt the built-in set. + // priority-list change. A matched plugin can supersede its declared fallback providers. for (FileSystemProvider provider : providers) { if (StorageRegistry.Provider.byName(provider.name()).isPresent()) { continue; } - FileSystemProperties bound = tryBindUnlisted(provider, properties, useGuess, probeView); + FileSystemProperties bound = tryBindUnlisted(provider, properties, + useGuess && !fallbacks.contains(provider.name()), probeView); if (bound != null) { return bound; } @@ -270,8 +308,10 @@ public FileSystemProperties bindPrimary(Map properties) { * default HDFS binding is inserted at index 0 (fe-core's default-HDFS fallback). */ public List bindAll(Map properties) { + validateProperties(properties); boolean useGuess = !hasAnyExplicitFsSupport(properties); Map probeView = withProbeContext(properties); + Set fallbacks = fallbackProviders(properties, probeView, useGuess); List result = new ArrayList<>(); boolean ossHdfsMatched = false; boolean hdfsFamilyMatched = false; @@ -291,7 +331,7 @@ public List bindAll(Map properties) { if (meta == StorageRegistry.Provider.HDFS && hdfsFamilyMatched) { continue; } - if (provider.supportsExplicit(properties) || (useGuess && provider.supportsGuess(probeView))) { + if (matches(provider, properties, probeView, useGuess, fallbacks)) { result.add(provider.bind(properties)); if (meta == StorageRegistry.Provider.OSS_HDFS) { ossHdfsMatched = true; @@ -310,7 +350,8 @@ public List bindAll(Map properties) { if (StorageRegistry.Provider.byName(provider.name()).isPresent()) { continue; } - FileSystemProperties bound = tryBindUnlisted(provider, properties, useGuess, probeView); + FileSystemProperties bound = tryBindUnlisted(provider, properties, + useGuess && !fallbacks.contains(provider.name()), probeView); if (bound != null) { result.add(bound); } @@ -379,7 +420,7 @@ public static Map withProbeContext(Map propertie private FileSystemProvider providerByName(String name) { for (FileSystemProvider p : providers) { - if (name.equalsIgnoreCase(p.name())) { + if (p.name().equalsIgnoreCase(name)) { return p; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java index 29feec06d54b2e..af6a47c1f2b881 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/fs/FileSystemPluginManagerTest.java @@ -123,6 +123,157 @@ public void bindAll_returnsEmptyListWhenNoProviderSupports() { // modules: runtime dependencies removed in Phase 4 P4.1"). End-to-end binding against the real // providers is therefore covered by P1-T06 (docker / full plugin classpath), not here. + @Test + public void specificPluginSupersedesFallbackInPrimaryAndAllBindings() throws Exception { + for (boolean specificFirst : new boolean[] {false, true}) { + FileSystemPluginManager manager = new FileSystemPluginManager(); + FileSystemProvider specific = routingProvider("custom", Set.of("s3")); + FileSystemProvider general = routingProvider("S3", Set.of()); + manager.registerProvider(specificFirst ? specific : general); + manager.registerProvider(specificFirst ? general : specific); + Map props = Map.of("custom.match", "true", "S3.match", "true"); + Assertions.assertEquals("custom", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("custom", manager.bindAll(props).get(0).providerName()); + Assertions.assertEquals("custom", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + + // A nonmatching specific plugin must not remove the general fallback. + Assertions.assertEquals("S3", manager.bindPrimary(Map.of("S3.match", "true")).providerName()); + } + } + + @Test + public void explicitFlagsWinAndMultipleExplicitFlagsRemainSupported() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("custom", Set.of("s3"))); + manager.registerProvider(routingProvider("S3", Set.of())); + Map props = new HashMap<>(Map.of("custom.match", "true", "S3.match", "true")); + props.put("fs.s3.support", "true"); + Assertions.assertEquals("S3", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("S3", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + props.put("fs.custom.support", "true"); + Assertions.assertEquals(2, manager.bindAll(props).size()); + } + + @Test + public void objectStorageGuessPreservesDefaultHdfsAndExplicitFlagsDisableIt() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("S3", Set.of())); + manager.registerProvider(routingProvider("HDFS", Set.of())); + List bindings = manager.bindAll(Map.of("provider", "s3", "S3.match", "true")); + Assertions.assertEquals(2, bindings.size()); + Assertions.assertEquals("HDFS", bindings.get(0).providerName()); + Assertions.assertEquals("S3", bindings.get(1).providerName()); + Assertions.assertEquals(1, manager.bindAll(Map.of("fs.s3.support", "true")).size()); + } + + @Test + public void providerNameDoesNotOverrideExistingMatchOrder() throws Exception { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("S3", Set.of())); + manager.registerProvider(routingProvider("HDFS", Set.of())); + Map props = new HashMap<>(Map.of( + "provider", "S3", "S3.match", "true", "HDFS.match", "true")); + Assertions.assertEquals("HDFS", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(2, manager.bindAll(props).size()); + Assertions.assertEquals("HDFS", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + + props.put("fs.hdfs.support", "true"); + Assertions.assertEquals("HDFS", manager.bindPrimary(props).providerName()); + Assertions.assertEquals(1, manager.bindAll(props).size()); + Assertions.assertEquals("HDFS", Assertions.assertThrows(UnsupportedOperationException.class, + () -> manager.createFileSystem(props)).getMessage()); + } + + @Test + public void unrelatedGuessesAreNotEvaluatedBeforePrimaryMatch() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("S3", Set.of())); + manager.registerProvider(new FileSystemProvider() { + @Override + public String name() { + return "unused"; + } + + @Override + public boolean supports(Map properties) { + throw new AssertionError("A later unrelated provider must not be probed"); + } + + @Override + public FileSystem create(Map properties) { + throw new AssertionError("A later unrelated provider must not be created"); + } + + @Override + public boolean supportsGuess(Map properties) { + throw new AssertionError("A later unrelated provider must not be probed"); + } + }); + Assertions.assertEquals("S3", manager.bindPrimary(Map.of("S3.match", "true")).providerName()); + } + + @Test + public void nonSelectedPluginStillValidatesItsOwnedProperties() { + FileSystemPluginManager manager = new FileSystemPluginManager(); + manager.registerProvider(routingProvider("custom", Set.of())); + manager.registerProvider(routingProvider("S3", Set.of())); + Map props = Map.of("provider", "S3", "custom.invalid", "true"); + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.bindPrimary(props)); + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.bindAll(props)); + Assertions.assertThrows(IllegalArgumentException.class, () -> manager.createFileSystem(props)); + } + + private static FileSystemProvider routingProvider(String name, Set fallbacks) { + return new FileSystemProvider() { + @Override + public Set fallbackProviders() { + return fallbacks; + } + + @Override + public void validateProperties(Map properties) { + if (properties.containsKey(name + ".invalid")) { + throw new IllegalArgumentException("Invalid " + name + " properties"); + } + } + + @Override + public boolean supportsExplicit(Map properties) { + return "true".equals(properties.get("fs." + name.toLowerCase(java.util.Locale.ROOT) + ".support")); + } + + @Override + public boolean supportsGuess(Map properties) { + return properties.containsKey(name + ".match"); + } + + @Override + public boolean supports(Map properties) { + return supportsExplicit(properties) || supportsGuess(properties); + } + + @Override + public FileSystemProperties bind(Map properties) { + return new FakeFsProps(name); + } + + @Override + public FileSystem create(Map properties) { + throw new UnsupportedOperationException(name); + } + + @Override + public String name() { + return name; + } + }; + } + // ---- helpers ---- private static FileSystemProvider bindingProvider( diff --git a/fe/fe-filesystem/README.md b/fe/fe-filesystem/README.md index eb8e34ff06631a..6097357c5bd606 100644 --- a/fe/fe-filesystem/README.md +++ b/fe/fe-filesystem/README.md @@ -141,6 +141,23 @@ At runtime, `DirectoryPluginRuntimeManager` creates an isolated `ClassLoader` pe `software.amazon.awssdk.*`, `org.apache.hadoop.*`) are loaded parent-first to avoid cross- ClassLoader cast failures. +### Overlapping provider matches + +The binding registry preserves the existing provider order and `fs..support` flag semantics. +It does not treat a `provider` property as a global override; each plugin owns its interpretation +of that property. A provider can declare `fallbackProviders()` to suppress general-purpose +heuristic matches when it also matches. This applies to both primary and multi-provider binding; +explicit filesystem flags are never suppressed. Providers without this declaration retain their +existing multi-match behavior. The default HDFS binding and HDFS URI validation are preserved. + +`validateProperties()` runs on every loaded provider before selection, so invalid provider-owned +credentials cannot silently disappear when another plugin is chosen. It must be deterministic and +perform no I/O. Forced provider binding validates with the effective provider name too. + +Filesystem plugin API 2.0 includes these two default hooks; plugins must be rebuilt +against the matching host API. Existing providers retain their behavior until they +opt into these hooks. + ### S3 Delegation Pattern Several cloud providers (OSS, COS, OBS) are S3-compatible. Instead of duplicating the S3 diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java index 7e4859dbb147a5..4e0cdb1347b0b9 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java +++ b/fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/FileSystemProvider.java @@ -44,6 +44,23 @@ */ public interface FileSystemProvider

extends PluginFactory { + /** + * Validate provider-owned properties before routing, including combinations that would otherwise + * select another provider and silently ignore these properties. Must be deterministic and do no I/O. + * Called for every loaded provider, even when it does not match the map. + */ + default void validateProperties(Map properties) { + } + + /** + * Names of general-purpose providers whose heuristic matches this provider supersedes when it + * also matches. Explicit selections are never suppressed. Names are case-insensitive; the + * relationship must be acyclic. Empty by default to preserve existing multi-provider bindings. + */ + default Set fallbackProviders() { + return Collections.emptySet(); + } + /** * Returns true if this provider can handle the given properties. * Must be cheap (no network calls) and deterministic. diff --git a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt index 819b07ca74430e..c489f587701c84 100644 --- a/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt +++ b/fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt @@ -13,11 +13,13 @@ org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.exten org.apache.doris.filesystem.spi.FileSystemProvider#create(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem org.apache.doris.filesystem.spi.FileSystemProvider#createUntyped(org.apache.doris.filesystem.properties.FileSystemProperties):org.apache.doris.filesystem.FileSystem org.apache.doris.filesystem.spi.FileSystemProvider#description():java.lang.String +org.apache.doris.filesystem.spi.FileSystemProvider#fallbackProviders():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#name():java.lang.String org.apache.doris.filesystem.spi.FileSystemProvider#sensitivePropertyKeys():java.util.Set org.apache.doris.filesystem.spi.FileSystemProvider#supports(java.util.Map):boolean org.apache.doris.filesystem.spi.FileSystemProvider#supportsExplicit(java.util.Map):boolean org.apache.doris.filesystem.spi.FileSystemProvider#supportsGuess(java.util.Map):boolean +org.apache.doris.filesystem.spi.FileSystemProvider#validateProperties(java.util.Map):void org.apache.doris.filesystem.spi.ObjFileSystem#capability(java.lang.Class):java.util.Optional org.apache.doris.filesystem.spi.ObjFileSystem#close():void org.apache.doris.filesystem.spi.ObjFileSystem#completeMultipartUpload(java.lang.String,java.lang.String,java.util.Map):void diff --git a/fe/fe-filesystem/pom.xml b/fe/fe-filesystem/pom.xml index a8368afa785571..faf427c0dc42c3 100644 --- a/fe/fe-filesystem/pom.xml +++ b/fe/fe-filesystem/pom.xml @@ -53,7 +53,7 @@ under the License. bumping all five families. See plan-doc/designs/2026-07-29-plugin-api-version-check-design.md. --> - 1.0 + 2.0 From 27707c699948f43df292e6e0d9b4ffebce8b78aa Mon Sep 17 00:00:00 2001 From: laihui <1353307710@qq.com> Date: Fri, 23 Oct 2026 21:20:13 +0800 Subject: [PATCH 47/47] [refactor](storage) Focus native GCP IAM review on provider support ### What problem does this PR solve? Related PR: #68540 Problem Summary: Remove three AWS-only suite copies from object_storage_iam_p0; the existing aws_iam_role_p0 suites provide the same coverage. Preserve the generic failed-rename regression from the extracted ALTER prerequisite alongside the GCP-specific regression. Document the native OAuth endpoint boundary. The feature branch now includes the independently reviewable ALTER and routing prerequisite commits without rewriting its published history. ### Release note None; production behavior is unchanged by this review split. ### Check List (For Author) - Test: Verified production sources are unchanged from pre-split HEAD, both prerequisite branches have no new GCP implementation/protocol dependencies, original AWS suites are byte-identical to master, clang-format 16 passed for the retained Cloud regression, and git diff --check passed. Full FE/Cloud validation remains blocked by local Thrift/thirdparty toolchain prerequisites. - Behavior changed: No production changes; duplicate AWS suites removed. - Does this need documentation: Filesystem README clarifies supported endpoints. --- cloud/test/meta_service_test.cpp | 23 +++ fe/fe-filesystem/README.md | 6 + .../test_catalog_instance_profile.groovy | 134 ------------------ .../test_catalog_with_role.groovy | 123 ---------------- ...tables_catalog_credentials_provider.groovy | 110 -------------- 5 files changed, 29 insertions(+), 367 deletions(-) delete mode 100644 regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy delete mode 100644 regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy delete mode 100644 regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp index 59cf5b0dfe199e..8442eee2f6e0bf 100644 --- a/cloud/test/meta_service_test.cpp +++ b/cloud/test/meta_service_test.cpp @@ -820,6 +820,29 @@ TEST(MetaServiceTest, AlterS3StorageVaultTest) { } } + // Retain the prerequisite regression: an unpaired static key must not commit a rename. + { + AlterObjStoreInfoRequest req; + req.set_cloud_unique_id("test_cloud_unique_id"); + req.set_op(AlterObjStoreInfoRequest::ALTER_S3_VAULT); + req.mutable_vault()->set_name(vault_name); + req.mutable_vault()->set_alter_name("rename_must_not_commit"); + req.mutable_vault()->mutable_obj_info()->set_ak("unpaired-key"); + + brpc::Controller cntl; + AlterObjStoreInfoResponse res; + meta_service->alter_storage_vault( + reinterpret_cast<::google::protobuf::RpcController*>(&cntl), &req, &res, nullptr); + ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT) << res.status().msg(); + + InstanceInfoPB stored_instance; + get_test_instance(stored_instance); + ASSERT_EQ(stored_instance.storage_vault_names(0), vault_name); + StorageVaultPB stored_vault; + get_test_vault("2", stored_vault); + ASSERT_EQ(stored_vault.name(), vault_name); + } + // A rejected credential update must not commit a rename that was applied // earlier in the request. { diff --git a/fe/fe-filesystem/README.md b/fe/fe-filesystem/README.md index 2c5df636e4f3a6..95d5f8b4216e1c 100644 --- a/fe/fe-filesystem/README.md +++ b/fe/fe-filesystem/README.md @@ -173,6 +173,12 @@ matching BE/Cloud support. Filesystem plugins cannot independently extend the re The filesystem plugin API is 5.0 and connector plugin API is 15.0 because shared authentication types changed. Plugins must be rebuilt against the matching host APIs. +Native OAuth accepts only the Google Storage service-base HTTPS endpoints recognized by +`GcsEndpoint` (port 443, without a bucket, path, or query). Arbitrary S3-compatible endpoints +and raw `fs.gs.auth.*` overrides are not supported in native mode. HMAC authentication retains +its existing endpoint behavior. BE error-log downloads use IAM `signBlob`; FE presigned +uploads remain unsupported with native OAuth. + ### Overlapping provider matches The binding registry preserves the existing provider order and `fs..support` flag semantics. diff --git a/regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy b/regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy deleted file mode 100644 index a617fe34ffbb24..00000000000000 --- a/regression-test/suites/object_storage_iam_p0/test_catalog_instance_profile.groovy +++ /dev/null @@ -1,134 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -import com.google.common.base.Strings; -import org.apache.doris.regression.util.ObjectStorageIamTestUtils - -suite("test_catalog_instance_profile") { - if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { - logger.info("skip ${name} because objectStorageIamProvider is not AWS") - return - } - - if (Strings.isNullOrEmpty(context.config.otherConfigs.get("hiveGlueInstanceProfileQueryTableName"))) { - return - } - sql """ ADMIN SET FRONTEND CONFIG ("aws_credentials_provider_version"="v2"); """ - String hiveGlueQueryTableName = context.config.otherConfigs.get("hiveGlueInstanceProfileQueryTableName") - String hiveGlueExpectCounts = context.config.otherConfigs.get("hiveGlueInstanceProfileExpectCounts") - String icebergFsQueryTableName = context.config.otherConfigs.get("icebergFsInstanceProfileQueryTableName") - String icebergFsExpectCounts = context.config.otherConfigs.get("icebergFsInstanceProfileExpectCounts") - String icebergFsWarehouse = context.config.otherConfigs.get("icebergFsInstanceProfileWarehouse") - String region = context.config.otherConfigs.get("awsInstanceProfileRegion") - //query method - def createCatalogAndQuery = { catalogProps, catalogName, queryTableName, expectCounts -> - sql """drop catalog if exists ${catalogName}""" - sql """ - ${catalogProps} - """ - def result = sql """ - select count(1) from ${catalogName}.${queryTableName}; - """ - println("result: ${result}") - def countValue = result[0][0] - assertTrue(countValue == expectCounts.toInteger()) - sql """drop catalog if exists ${catalogName}""" - } - def assertCatalogAndQueryException = { catalogProps, catalogName, queryTableName -> - sql """drop catalog if exists ${catalogName}""" - sql """ - ${catalogProps} - """ - boolean failed = false - try { - sql """ - switch ${catalogName}; - """ - sql """ - show databases; - """ - sql """ - select count(1) from ${catalogName}.${queryTableName}; - """ - }catch (Exception e){ - failed = true - } - assertTrue(failed) - } - String hiveGlueCatalogProps = """ - create catalog hive_glue_catalog_instance_profile properties( - "type"="hms", - "hive.metastore.type"="glue", - "glue.region"="${region}", - "glue.endpoint" = "https://glue.${region}.amazonaws.com" - ); - """ - createCatalogAndQuery(hiveGlueCatalogProps, "hive_glue_catalog_instance_profile", hiveGlueQueryTableName, hiveGlueExpectCounts) - hiveGlueCatalogProps = """ - create catalog hive_glue_catalog_instance_profile properties( - "type"="hms", - "hive.metastore.type"="glue", - "glue.credentials_provider_type"="INSTANCE_PROFILE", - "glue.region"="${region}", - "glue.endpoint" = "https://glue.${region}.amazonaws.com" - ); - """ - createCatalogAndQuery(hiveGlueCatalogProps, "hive_glue_catalog_instance_profile", hiveGlueQueryTableName, hiveGlueExpectCounts) - hiveGlueCatalogProps = """ - create catalog hive_glue_catalog_instance_profile properties( - "type"="hms", - "hive.metastore.type"="glue", - "glue.credentials_provider_type"="CONTAINER", - "glue.region"="${region}", - "glue.endpoint" = "https://glue.${region}.amazonaws.com" - ); - """ - assertCatalogAndQueryException(hiveGlueCatalogProps,"hive_glue_catalog_instance_profile", hiveGlueQueryTableName) - String icebergFsCatalogProps = """ - create catalog iceberg_fs_catalog_instance_profile properties( - "type"="iceberg", - "warehouse"="${icebergFsWarehouse}", - "iceberg.catalog.type"="hadoop", - "s3.region" = "${region}", - "s3.endpoint" = "https://s3.${region}.amazonaws.com" - ); - """ - createCatalogAndQuery(icebergFsCatalogProps, "iceberg_fs_catalog_instance_profile", icebergFsQueryTableName, icebergFsExpectCounts) - icebergFsCatalogProps = """ - create catalog iceberg_fs_catalog_instance_profile properties( - "type"="iceberg", - "warehouse"="${icebergFsWarehouse}", - "iceberg.catalog.type"="hadoop", - "s3.credentials_provider_type"="INSTANCE_PROFILE", - "s3.region" = "${region}", - "s3.endpoint" = "https://s3.${region}.amazonaws.com" - ); - """ - createCatalogAndQuery(icebergFsCatalogProps, "iceberg_fs_catalog_instance_profile", icebergFsQueryTableName, icebergFsExpectCounts) - icebergFsCatalogProps = """ - create catalog iceberg_fs_catalog_instance_profile properties( - "type"="iceberg", - "warehouse"="${icebergFsWarehouse}", - "iceberg.catalog.type"="hadoop", - "s3.credentials_provider_type"="CONTAINER", - "s3.region" = "${region}", - "s3.endpoint" = "https://s3.${region}.amazonaws.com" - ); - """ - assertCatalogAndQueryException(icebergFsCatalogProps,"iceberg_fs_catalog_instance_profile",icebergFsQueryTableName) - -} diff --git a/regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy b/regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy deleted file mode 100644 index d6eda63c8ef3b0..00000000000000 --- a/regression-test/suites/object_storage_iam_p0/test_catalog_with_role.groovy +++ /dev/null @@ -1,123 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -import com.google.common.base.Strings; -import org.apache.doris.regression.util.ObjectStorageIamTestUtils - -suite("test_catalog_with_role") { - if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { - logger.info("skip ${name} because objectStorageIamProvider is not AWS") - return - } - if (Strings.isNullOrEmpty(context.config.otherConfigs.get("awsLakesRoleArn"))) { - logger.info("skip ${name} case, because awsRoleArn is null or empty") - return - } - //query method - def createCatalogAndQuery = { catalogProps, catalogName, queryTableName, expectCounts -> - sql """drop catalog if exists ${catalogName}""" - sql """ - ${catalogProps} - """ - def result = sql """ - select count(1) from ${catalogName}.${queryTableName}; - """ - println("result: ${result}") - def countValue = result[0][0] - assertTrue(countValue == expectCounts.toInteger()) - sql """drop catalog if exists ${catalogName}""" - } - String suiteName = "test_catalog_with_role" - String awsLakersRoleArn = context.config.otherConfigs.get("awsLakesRoleArn") - String awsLakesRegion = context.config.otherConfigs.get("awsLakesRegion") - String icebergFSOnS3CatalogWarehouse = context.config.otherConfigs.get("icebergFSOnS3CatalogWarehouse") - //String hmsMetastoreHost = context.config.hmsMetastoreHost - //String icebergHmsTableName = context.config.icebergHmsTableName - String icebergGlueTableName = context.config.otherConfigs.get("icebergGlueTableName") - - String icebergFSOnS3TableName = context.config.otherConfigs.get("icebergFSOnS3TableName") - //String hiveHmsTableName = context.config.hiveHmsTableName - String hiveGlueTableName = context.config.otherConfigs.get("hiveGlueTableName") - String expectCounts = context.config.otherConfigs.get("awsLakesRoleArnQueryExpectCounts") - - String awsS3Property = """ - "s3.region" = "${awsLakesRegion}", - "s3.role_arn" = "${awsLakersRoleArn}" - """ - String awsGlueProperties = """ - "glue.region"="${awsLakesRegion}", - "glue.endpoint" = "https://glue.${awsLakesRegion}.amazonaws.com", - "glue.role_arn" = "${awsLakersRoleArn}" - """ - // start test - String catalogName = "${suiteName}_iceberg_hadoop_on_s3" - //1. iceberg hadoop catalog on s - String icebergFSCatalogProps = """ - create catalog if not exists ${catalogName} properties( - "type"="iceberg", - "iceberg.catalog.type"="hadoop", - "warehouse"="${icebergFSOnS3CatalogWarehouse}", - ${awsS3Property} - ); - """ - createCatalogAndQuery(icebergFSCatalogProps, catalogName, icebergFSOnS3TableName, expectCounts) - catalogName = "${suiteName}_iceberg_glue" - String icebergGlueCatalogProps = """ - create catalog if not exists ${catalogName} properties( - "type"="iceberg", - "iceberg.catalog.type"="glue", - ${awsGlueProperties} - ); - """ - createCatalogAndQuery(icebergGlueCatalogProps, catalogName, icebergGlueTableName, expectCounts) - catalogName = "${suiteName}_hive_glue" - String hiveGlueCatalogProps = """ - create catalog if not exists ${catalogName} properties ( - "type"="hms", - "hive.metastore.type" = "glue", - ${awsGlueProperties} - ); - """ - createCatalogAndQuery(hiveGlueCatalogProps, catalogName, hiveGlueTableName, expectCounts) -/*********** HMS ***************/ -// The following HMS-related cases are commented out because -// they require a Docker environment for the Hive Metastore service. - -/* -String catalogName = "${suiteName}iceberg_hive_on_s3" -String icebergHmsCatalogProps = """ - create catalog if not exists ${catalogName} properties ( - "type" = "iceberg", - "iceberg.catalog.type" = "hive", - "hive.metastore.uris" = "thrift://${hmsMetastoreHost}:9083", - ${awsS3Property} - ); -""" -createCatalogAndQuery(icebergHmsCatalogProps, catalogName, icebergHmsTableName, expectCounts) - -String hiveHmsCatalogProps = """ - create catalog if not exists ${catalogName} properties ( - "type" = "hms", - "hive.metastore.uris" = "thrift://${hmsMetastoreHost}:9083", - ${awsS3Property} - ); -""" -createCatalogAndQuery(hiveHmsCatalogProps, catalogName, hiveHmsTableName, expectCounts) -*/ - - -} diff --git a/regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy b/regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy deleted file mode 100644 index bbef874246c836..00000000000000 --- a/regression-test/suites/object_storage_iam_p0/test_iceberg_s3tables_catalog_credentials_provider.groovy +++ /dev/null @@ -1,110 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -import com.google.common.base.Strings; -import org.apache.doris.regression.util.ObjectStorageIamTestUtils - -suite("test_iceberg_s3tables_catalog_credentials_provider") { - if (!ObjectStorageIamTestUtils.isProvider(context.config.otherConfigs, "AWS")) { - logger.info("skip ${name} because objectStorageIamProvider is not AWS") - return - } - - List requiredConfigNames = [ - "icebergS3TablesCatalog1Props", - "icebergS3TablesCatalog2Props", - "icebergS3TablesCatalog3Props", - "icebergS3TablesPartitionDbTableName", - "icebergS3TablesNonPartitionDbTableName", - "icebergS3TablesExpectCounts" - ] - if (requiredConfigNames.any { Strings.isNullOrEmpty(context.config.otherConfigs.get(it)) }) { - return - } - - sql """ ADMIN SET FRONTEND CONFIG ("aws_credentials_provider_version"="v2"); """ - - String catalog1Props = context.config.otherConfigs.get("icebergS3TablesCatalog1Props") - String catalog2Props = context.config.otherConfigs.get("icebergS3TablesCatalog2Props") - String catalog3Props = context.config.otherConfigs.get("icebergS3TablesCatalog3Props") - String partitionDbTableName = context.config.otherConfigs.get("icebergS3TablesPartitionDbTableName") - String nonPartitionDbTableName = context.config.otherConfigs.get("icebergS3TablesNonPartitionDbTableName") - String expectCounts = context.config.otherConfigs.get("icebergS3TablesExpectCounts") - String expectedWebIdentityError = "Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set." - - def createCatalogAndQueryTables = { - catalogProps, catalogName, partitionQueryDbTableName, nonPartitionQueryDbTableName, expectedCounts -> - sql """drop catalog if exists ${catalogName}""" - try { - sql """ - create catalog ${catalogName} properties ( - ${catalogProps} - ); - """ - def partitionResult = sql """ - select count(1) from ${catalogName}.${partitionQueryDbTableName}; - """ - def nonPartitionResult = sql """ - select count(1) from ${catalogName}.${nonPartitionQueryDbTableName}; - """ - def partitionCount = partitionResult[0][0] - def nonPartitionCount = nonPartitionResult[0][0] - assertTrue(partitionCount == expectedCounts.toInteger()) - assertTrue(nonPartitionCount == expectedCounts.toInteger()) - assertTrue(partitionCount == nonPartitionCount) - } finally { - sql """drop catalog if exists ${catalogName}""" - } - } - - def createCatalogAndShowDatabasesException = { catalogProps, catalogName, expectedMessage -> - sql """drop catalog if exists ${catalogName}""" - try { - sql """ - create catalog ${catalogName} properties ( - ${catalogProps} - ); - """ - sql """switch ${catalogName};""" - try { - sql """show databases;""" - assertTrue(false) - } catch (Exception e) { - assertTrue(e.getMessage().contains(expectedMessage)) - } - } finally { - sql """drop catalog if exists ${catalogName}""" - } - } - - createCatalogAndQueryTables( - catalog1Props, - "iceberg_s3tables_catalog_1", - partitionDbTableName, - nonPartitionDbTableName, - expectCounts) - createCatalogAndQueryTables( - catalog2Props, - "iceberg_s3tables_catalog_2", - partitionDbTableName, - nonPartitionDbTableName, - expectCounts) - createCatalogAndShowDatabasesException( - catalog3Props, - "iceberg_s3tables_catalog_3", - expectedWebIdentityError) -}