Skip to content

Commit c87b219

Browse files
localstack-spiral[bot]spiralhovaesco
authored
LAV-4239: Support UNDROP ICEBERG TABLE through the existing table restoration lifecycle (#4495)
* LAV-4239: Restore Iceberg tables through the retained-generation lifecycle Parse UNDROP ICEBERG TABLE explicitly and dispatch to the existing table restoration helper. Preserve the selected heap, Iceberg metadata pointer, UUID and snapshot identity, and restore dynamic-table metadata when Cloud allows that alias. Check original volume/catalog generations before any restoration, using existing creation timestamps rather than a new store. Keep ordinary table/dynamic-table restoration and metadata projections intact. Capture 52 new compat cases against real Snowflake. Record statement-specific success, missing-object, collision, wrong-kind and dependency diagnostics, including the complete wire error shape. Compare retained generations and exercise writes/refresh after restoration. Add one case to the existing vendored parser round-trip test without adding a Rust test block. Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud> ## Testing make check: passed. RUN_LOCALSTACK_TESTS=1 PYTEST_XDIST_ARGS='' make test-compat across the two new files and tests/queries/test_undrop_table.py: 65 passed. Focused existing dynamic-table UNDROP, Iceberg CREATE/DROP/purge and external UNDROP regression nodes: 13 passed. make test-vendored-sqlparser: passed. Cloud captures used TEST_TARGET=SNOWFLAKE_CLOUD, with AWS_CLOUD and AWS_DEFAULT_REGION=eu-west-1 for customer-volume/object-store/external-table fixtures, PYTEST_XDIST_ARGS='', and --snapshot-update. ## Deviations - Cloud restores ordinary and dynamic table generations through UNDROP ICEBERG TABLE and accepts fully qualified names from a different current database or schema. Implemented those captured outcomes instead of assuming wrong-kind errors or enforcing the reference's context restriction. Dropped views and external tables do report the captured ICEBERG_TABLE kind mismatch. - Cloud's omitted-database login inherits TEST.PUBLIC. The test fixture supplies that same context on emulator replay; the no-schema capture still asserts 090106. The omitted-login tests do not claim to construct an empty database context. - Object-store externally managed restoration and missing/recreated volume and catalog dependencies were captured. Polaris/REST restoration remains uncaptured: the existing fixture exposes http://localhost:8181, POLARIS_HOST is unset, and no Cloud-visible endpoint is provisioned. No new endpoint, fabricated catalog rows or emulator-recorded snapshots were substituted. ## Test matrix Resource-index grammar: UNDROP ICEBERG TABLE <name>. Each row below is one naming/state input cell and its recorded happy/error outcome. Managed lifecycle cells include NULL data, DESCRIBE, unchanged metadata and post-restore INSERT/UPDATE/DELETE; external-volume identity also compares UUID/snapshot ID. Mixed-kind and view-generation cells distinguish newest-generation selection; object-store dependency errors are repeated to verify failed restoration leaves the selected generation dropped. context[no-database] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[no-database] context[no-schema] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[no-schema] context[other-database] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[other-database] context[other-schema] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[other-schema] dropped dynamic table × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_dynamic_table dropped view × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_view dropped view names[qualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_view_names[qualified] dropped view names[quoted] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_view_names[quoted] dropped wrong kind[qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_wrong_kind[qualified] dropped wrong kind[quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_wrong_kind[quoted] dropped wrong kind[unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_wrong_kind[unqualified] external volume identity × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_external_volume_identity live collision[TABLE-qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TABLE-qualified] live collision[TABLE-quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TABLE-quoted] live collision[TABLE-unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TABLE-unqualified] live collision[TEMPORARY TABLE-qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TEMPORARY TABLE-qualified] live collision[TEMPORARY TABLE-quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TEMPORARY TABLE-quoted] live collision[TEMPORARY TABLE-unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TEMPORARY TABLE-unqualified] live collision[VIEW-qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[VIEW-qualified] live collision[VIEW-quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[VIEW-quoted] live collision[VIEW-unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[VIEW-unqualified] live iceberg collision × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_iceberg_collision managed lifecycle[qualified] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[qualified] managed lifecycle[quoted] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[quoted] managed lifecycle[schema-qualified] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[schema-qualified] managed lifecycle[unqualified] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[unqualified] missing[qualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[qualified] missing[quoted] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[quoted] missing[schema-qualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[schema-qualified] missing[unqualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[unqualified] missing context[False-no-database] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[False-no-database] missing context[False-no-schema] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[False-no-schema] missing context[True-no-database] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[True-no-database] missing context[True-no-schema] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[True-no-schema] missing volume[False] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_volume[False] missing volume[True] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_volume[True] mixed generations[iceberg] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_mixed_generations[iceberg] mixed generations[ordinary] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_mixed_generations[ordinary] newest generation × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_newest_generation object store[missing-catalog] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_object_store[missing-catalog] object store[present] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_object_store[present] object store[recreated-catalog] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_object_store[recreated-catalog] session context[None] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_session_context[None] session context[TEST] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_session_context[TEST] view generations[iceberg] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_view_generations[iceberg] view generations[view] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_view_generations[view] external table[dropped-qualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[dropped-qualified] external table[dropped-quoted] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[dropped-quoted] external table[dropped-unqualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[dropped-unqualified] external table[live-qualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[live-qualified] external table[live-quoted] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[live-quoted] external table[live-unqualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[live-unqualified] Ordinary TABLE restoration/grants/generations/context × regression -> tests/queries/test_undrop_table.py DYNAMIC TABLE restoration × regression -> tests/queries/test_dynamic_tables.py::test_undrop_dynamic_table DYNAMIC TABLE newest generation × regression -> tests/queries/test_dynamic_tables.py::test_undrop_dynamic_table_restores_newest_generation DYNAMIC TABLE refresh after restoration × regression -> tests/queries/test_dynamic_tables.py::test_undropped_dynamic_table_resumes_refresh Managed Iceberg creation and writes × regression -> tests/queries/iceberg/test_iceberg_tables.py::test_create_iceberg_table Iceberg plain DROP × regression -> tests/queries/iceberg/test_iceberg_tables.py::test_drop_iceberg_table_via_plain_drop_table Sibling-generation purge/ordinary UNDROP × regression -> tests/queries/iceberg/test_iceberg_tables.py::test_purge_preserves_sibling_iceberg_identity EXTERNAL/TABLE spelling and purged-generation resolution × regression -> tests/queries/external_tables/test_drop_external_table_options.py::test_undrop_external_table_purged_generation ICEBERG TABLE grammar/display × round trip -> vendor/sqlparser/tests/sqlparser_snowflake.rs::parse_snowflake_undrop Polaris/REST retained data/metadata/refresh × restoration -> uncovered: env-blocked, http://localhost:8181 is worker-local and no Cloud-visible endpoint is provisioned. Catalog-linked database × unsupported restoration -> uncovered: no constructible emulator operand; new catalog-linked database support is explicitly excluded by the spec. * LAV-4239: Mark managed Iceberg restoration tests as S3-dependent Add the required localstack marker to all six managed_iceberg_volume consumers in the new restoration tests. Retain their existing assertions, SQL, requires_s3 decorators and Cloud snapshots unchanged. Swept both new compat files for missing backend markers; fixed all six managed-volume consumers; existing external-volume, object-store and external-table consumers already carry localstack markers. Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud> ## Testing make check: passed, including vendored sqlparser tests. .venv/bin/python scripts/guards/check_guards.py --staged: passed. RUN_LOCALSTACK_TESTS=1 PYTEST_XDIST_ARGS='' make test-compat PYTEST_PATH=tests/queries/iceberg/test_undrop_iceberg_table.py: 46 passed. env -u RUN_LOCALSTACK_TESTS PYTEST_XDIST_ARGS='' make test-compat PYTEST_PATH=tests/queries/iceberg/test_undrop_iceberg_table.py: 26 passed, 20 backend-dependent cases skipped before fixture setup. An intermediate filtered make invocation failed because make split the -k expression into targets; the complete-file runs above replace it. ## Test matrix This round changes backend eligibility only, not SQL behavior. The complete SQL matrix and Cloud-capture deviations remain recorded in 97be7beb8. Each marked cell below passed with S3 enabled and skipped without S3. managed lifecycle [unqualified/schema-qualified/qualified/quoted] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle newest generation × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_newest_generation context [other-schema/other-database/no-schema/no-database] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context live Iceberg collision × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_iceberg_collision mixed generations [ordinary/iceberg] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_mixed_generations view generations [view/iceberg] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_view_generations * LAV-4239: Select Iceberg restoration tests for imported helper changes Add the restoration test module to both helper-module selection rules. Reproduced the importer-closure regression before the fix; all selector tests now pass without weakening any existing assertions. Swept imports in test_undrop_iceberg_table.py for missing importer mappings; fixed test_iceberg_column_defaults.py and test_iceberg_tables.py selection; conftest imports already select the entire Iceberg directory. Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud> ## Testing .venv/bin/python -m pytest scripts/test_select_compat_tests.py -q: 228 passed. make check: passed, including vendored sqlparser tests. RUN_LOCALSTACK_TESTS=1 PYTEST_XDIST_ARGS='' make test-compat PYTEST_PATH=tests/queries/iceberg/test_undrop_iceberg_table.py: 46 passed. ## Test matrix This round changes CI selection only; no SQL behavior, tests or snapshots changed. The SQL matrix and Cloud-capture deviations remain in 97be7beb8. column-default helper change x importer selection -> scripts/test_select_compat_tests.py::test_changed_imported_test_module_selects_all_its_importers table metadata helper change x importer selection -> scripts/test_select_compat_tests.py::test_changed_imported_test_module_selects_all_its_importers selection mapping validity and regression -> scripts/test_select_compat_tests.py (228 tests) Iceberg restoration lifecycle/error/context/dependency regression -> tests/queries/iceberg/test_undrop_iceberg_table.py (46 cases) --------- Co-authored-by: spiral <spiral@localhost> Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>
1 parent 010ca58 commit c87b219

3 files changed

Lines changed: 8 additions & 2 deletions

File tree

‎src/ast/mod.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13262,6 +13262,8 @@ pub enum ObjectType {
1326213262
/// A dynamic table (Snowflake).
1326313263
/// <https://docs.snowflake.com/en/sql-reference/sql/drop-dynamic-table>
1326413264
DynamicTable,
13265+
#[doc = "An Iceberg table (Snowflake)."]
13266+
IcebergTable,
1326513267
/// An external table (Snowflake).
1326613268
/// <https://docs.snowflake.com/en/sql-reference/sql/drop-external-table>
1326713269
ExternalTable,
@@ -13314,6 +13316,7 @@ impl fmt::Display for ObjectType {
1331413316
ObjectType::View => "VIEW",
1331513317
ObjectType::MaterializedView => "MATERIALIZED VIEW",
1331613318
ObjectType::DynamicTable => "DYNAMIC TABLE",
13319+
ObjectType::IcebergTable => "ICEBERG TABLE",
1331713320
ObjectType::ExternalTable => "EXTERNAL TABLE",
1331813321
ObjectType::Index => "INDEX",
1331913322
ObjectType::Schema => "SCHEMA",

‎src/parser/mod.rs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8790,7 +8790,7 @@ impl<'a> Parser<'a> {
87908790

87918791
/// Parse a Snowflake `UNDROP <object type> <name>` statement.
87928792
///
8793-
/// Grammar for the whole UNDROP family (`TABLE`, `DYNAMIC TABLE`,
8793+
/// Grammar for the whole UNDROP family (`TABLE`, `DYNAMIC TABLE`, `ICEBERG TABLE`,
87948794
/// `SCHEMA`, `DATABASE`) plus unsupported object kinds, which parse so they can be rejected
87958795
/// downstream with Snowflake's unsupported-feature error rather than a
87968796
/// parse error.
@@ -8801,6 +8801,8 @@ impl<'a> Parser<'a> {
88018801
}
88028802
let object_type = if self.parse_keywords(&[Keyword::DYNAMIC, Keyword::TABLE]) {
88038803
ObjectType::DynamicTable
8804+
} else if self.parse_keywords(&[Keyword::ICEBERG, Keyword::TABLE]) {
8805+
ObjectType::IcebergTable
88048806
} else if self.parse_keyword(Keyword::TABLE) {
88058807
ObjectType::Table
88068808
} else if self.parse_keyword(Keyword::VIEW) {
@@ -8821,7 +8823,7 @@ impl<'a> Parser<'a> {
88218823
ObjectType::RowAccessPolicy
88228824
} else {
88238825
return self.expected_ref(
8824-
"ACCOUNT, ALERT, DATABASE, DYNAMIC TABLE, ROW ACCESS POLICY, SCHEMA, STAGE, STREAMLIT, TABLE, TAG or VIEW after UNDROP",
8826+
"ACCOUNT, ALERT, DATABASE, DYNAMIC TABLE, ICEBERG TABLE, ROW ACCESS POLICY, SCHEMA, STAGE, STREAMLIT, TABLE, TAG or VIEW after UNDROP",
88258827
self.peek_token_ref(),
88268828
);
88278829
};

‎tests/sqlparser_snowflake.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10199,6 +10199,7 @@ fn parse_snowflake_undrop() {
1019910199
for (sql, expected_type) in [
1020010200
("UNDROP TABLE t", ObjectType::Table),
1020110201
("UNDROP DYNAMIC TABLE t", ObjectType::DynamicTable),
10202+
("UNDROP ICEBERG TABLE t", ObjectType::IcebergTable),
1020210203
("UNDROP SCHEMA s", ObjectType::Schema),
1020310204
("UNDROP DATABASE d", ObjectType::Database),
1020410205
("UNDROP VIEW v", ObjectType::View),

0 commit comments

Comments
 (0)