[claudevdm]

Moves the BigLake REST catalog integration tests (RESTCatalogBLMSIT, AddFilesIT) from a legacy single-bucket catalog (gs://bucket warehouse) to a multiple-bucket catalog (bl://projects/P/catalogs/C warehouse), and adds coverage for tables that live outside the catalog's default location.

Test-only change: no IcebergIO production code is touched. IcebergIO passes catalog_properties straight to Iceberg's RESTCatalog and writes wherever the catalog places the table, so it needed no changes.

Why

Google now documents single-bucket catalogs as legacy and recommends multiple-bucket catalogs (docs). In a multiple-bucket catalog the catalog name is independent of any bucket, and namespaces can be placed in any of the catalog's allowed locations. The existing tests assumed "catalog id == bucket name == data bucket" throughout.

What changed

File	Change
BigLakeTestCatalog.java (new)	Single source of the BigLake test setup: warehouse, allowed locations, catalog properties, BigQuery 4-part table spec, and cleanup helpers. Reads beam.iceberg.biglake.warehouse and beam.iceberg.biglake.locations.
build.gradle	integrationTest sets those two properties. Defaults: bl://projects/apache-beam-testing/catalogs/beam-lakehouse-it, locations gs://beam-lakehouse-it,gs://beam-lakehouse-it-added-path. Override with -PbiglakeWarehouse / -PbiglakeLocations.
RESTCatalogBLMSIT.java	Uses the helper. Adds testWriteReadTableInAdditionalLocation. Overrides catalogCleanup to also delete table files.
AddFilesIT.java	Uses the helper. Separates "catalog" from "bucket holding the source parquet files and GCS notifications". Adds testBatchParquetImportToTableInAdditionalLocation.
IcebergCatalogBaseIT.java	readRecords is now protected. The GCS sweep in cleanUp no longer throws when the warehouse is a bare bucket.
IO_Iceberg_Integration_Tests.json	Bumped to trigger the suite.

New tests

• RESTCatalogBLMSIT.testWriteReadTableInAdditionalLocation: creates a namespace in the catalog's second bucket, writes with Managed.write(ICEBERG), and asserts the data files and table metadata are in that bucket.
• AddFilesIT.testBatchParquetImportToTableInAdditionalLocation: destination table in the second bucket, source parquet files in the default bucket. Verifies the files register and read back through both Iceberg and BigQueryIO.

BigLake behaviour that shaped the tests

• Placement is per namespace, not per table. BigLake rejects a table location outside its namespace's path, so the new tests create the namespace with a location property rather than calling withLocation on the table.
• Dropping a table does not delete its files, even with purge requested, and table directories get a random suffix. dropNamespacesAndFiles therefore records each table's location before dropping it and deletes those objects afterwards. Previously every BigLake test run left its files behind.
• Vended-credentials catalogs require the X-Iceberg-Access-Delegation: vended-credentials header. The helper always sends it; it is ignored by end-user-credential catalogs.
• Expected log noise: Failed to refresh storage credentials ... Invalid credentials endpoint: null from GCSFileIO on Iceberg 1.11.0. It comes from a background refresh that fails harmlessly; the storage client refreshes its token through a separate path.

Infra this depends on

• Catalog beam-lakehouse-it in apache-beam-testing, with default location gs://beam-lakehouse-it and additional location gs://beam-lakehouse-it-added-path.
• The catalog is in vended-credentials mode, so its service agent needs object access on both buckets, plus storage.buckets.get (e.g. roles/storage.bucketViewer) for the BigQuery read in the cross-bucket AddFiles test.

Running locally

./gradlew :sdks:java:io:iceberg:integrationTest \
  -PgcpProject=<project> -PgcpTempLocation=gs://<bucket>/temp \
  -PbiglakeWarehouse=bl://projects/<project>/catalogs/<catalog> \
  -PbiglakeLocations=gs://<default-location>,gs://<additional-location> \
  --tests '*RESTCatalogBLMSIT*' --tests '*AddFilesIT*'

The first location must be the catalog's default location and the second must be in a different bucket. A legacy gs://bucket warehouse is still accepted.

Testing

Run against an equivalent multiple-bucket catalog in a separate GCP project:

• Full RESTCatalogBLMSIT and AddFilesIT suites passed before the rebase onto current master.
• After the rebase: all 6 AddFilesIT tests, plus RESTCatalogBLMSIT.testWrite and testWriteReadTableInAdditionalLocation, passed, with no objects left in either bucket.
• RESTCatalogBLMSIT.testWrite and AddFilesIT.testBatchParquetImport also passed against a vended-credentials catalog, and testWrite against a legacy single-bucket catalog.

[claudevdm]

R: @ahmedabu98

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[ahmedabu98]

Lets rename everything (utility classes, comments, etc) from "BigLake" to "Lakehouse"

Besides that, this LGTM