Repository navigation
Commit c87b219
LAV-4239: Support UNDROP ICEBERG TABLE through the existing table restoration lifecycle (#4495)
* LAV-4239: Restore Iceberg tables through the retained-generation lifecycle
Parse UNDROP ICEBERG TABLE explicitly and dispatch to the existing table
restoration helper. Preserve the selected heap, Iceberg metadata pointer,
UUID and snapshot identity, and restore dynamic-table metadata when Cloud
allows that alias. Check original volume/catalog generations before any
restoration, using existing creation timestamps rather than a new store.
Keep ordinary table/dynamic-table restoration and metadata projections intact.
Capture 52 new compat cases against real Snowflake. Record statement-specific
success, missing-object, collision, wrong-kind and dependency diagnostics,
including the complete wire error shape. Compare retained generations and
exercise writes/refresh after restoration. Add one case to the existing
vendored parser round-trip test without adding a Rust test block.
Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>
## Testing
make check: passed.
RUN_LOCALSTACK_TESTS=1 PYTEST_XDIST_ARGS='' make test-compat across the two
new files and tests/queries/test_undrop_table.py: 65 passed.
Focused existing dynamic-table UNDROP, Iceberg CREATE/DROP/purge and external
UNDROP regression nodes: 13 passed.
make test-vendored-sqlparser: passed.
Cloud captures used TEST_TARGET=SNOWFLAKE_CLOUD, with AWS_CLOUD and
AWS_DEFAULT_REGION=eu-west-1 for customer-volume/object-store/external-table
fixtures, PYTEST_XDIST_ARGS='', and --snapshot-update.
## Deviations
- Cloud restores ordinary and dynamic table generations through UNDROP ICEBERG
TABLE and accepts fully qualified names from a different current database or
schema. Implemented those captured outcomes instead of assuming wrong-kind
errors or enforcing the reference's context restriction. Dropped views and
external tables do report the captured ICEBERG_TABLE kind mismatch.
- Cloud's omitted-database login inherits TEST.PUBLIC. The test fixture supplies
that same context on emulator replay; the no-schema capture still asserts
090106. The omitted-login tests do not claim to construct an empty database
context.
- Object-store externally managed restoration and missing/recreated volume and
catalog dependencies were captured. Polaris/REST restoration remains
uncaptured: the existing fixture exposes http://localhost:8181, POLARIS_HOST
is unset, and no Cloud-visible endpoint is provisioned. No new endpoint,
fabricated catalog rows or emulator-recorded snapshots were substituted.
## Test matrix
Resource-index grammar: UNDROP ICEBERG TABLE <name>. Each row below is one
naming/state input cell and its recorded happy/error outcome. Managed lifecycle
cells include NULL data, DESCRIBE, unchanged metadata and post-restore
INSERT/UPDATE/DELETE; external-volume identity also compares UUID/snapshot ID.
Mixed-kind and view-generation cells distinguish newest-generation selection;
object-store dependency errors are repeated to verify failed restoration
leaves the selected generation dropped.
context[no-database] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[no-database]
context[no-schema] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[no-schema]
context[other-database] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[other-database]
context[other-schema] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context[other-schema]
dropped dynamic table × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_dynamic_table
dropped view × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_view
dropped view names[qualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_view_names[qualified]
dropped view names[quoted] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_view_names[quoted]
dropped wrong kind[qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_wrong_kind[qualified]
dropped wrong kind[quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_wrong_kind[quoted]
dropped wrong kind[unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_dropped_wrong_kind[unqualified]
external volume identity × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_external_volume_identity
live collision[TABLE-qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TABLE-qualified]
live collision[TABLE-quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TABLE-quoted]
live collision[TABLE-unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TABLE-unqualified]
live collision[TEMPORARY TABLE-qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TEMPORARY TABLE-qualified]
live collision[TEMPORARY TABLE-quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TEMPORARY TABLE-quoted]
live collision[TEMPORARY TABLE-unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[TEMPORARY TABLE-unqualified]
live collision[VIEW-qualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[VIEW-qualified]
live collision[VIEW-quoted] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[VIEW-quoted]
live collision[VIEW-unqualified] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_collision[VIEW-unqualified]
live iceberg collision × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_iceberg_collision
managed lifecycle[qualified] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[qualified]
managed lifecycle[quoted] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[quoted]
managed lifecycle[schema-qualified] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[schema-qualified]
managed lifecycle[unqualified] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle[unqualified]
missing[qualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[qualified]
missing[quoted] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[quoted]
missing[schema-qualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[schema-qualified]
missing[unqualified] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing[unqualified]
missing context[False-no-database] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[False-no-database]
missing context[False-no-schema] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[False-no-schema]
missing context[True-no-database] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[True-no-database]
missing context[True-no-schema] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_context[True-no-schema]
missing volume[False] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_volume[False]
missing volume[True] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_missing_volume[True]
mixed generations[iceberg] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_mixed_generations[iceberg]
mixed generations[ordinary] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_mixed_generations[ordinary]
newest generation × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_newest_generation
object store[missing-catalog] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_object_store[missing-catalog]
object store[present] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_object_store[present]
object store[recreated-catalog] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_object_store[recreated-catalog]
session context[None] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_session_context[None]
session context[TEST] × happy/error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_session_context[TEST]
view generations[iceberg] × happy/boundary -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_view_generations[iceberg]
view generations[view] × error -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_view_generations[view]
external table[dropped-qualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[dropped-qualified]
external table[dropped-quoted] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[dropped-quoted]
external table[dropped-unqualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[dropped-unqualified]
external table[live-qualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[live-qualified]
external table[live-quoted] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[live-quoted]
external table[live-unqualified] × error -> tests/queries/external_tables/test_undrop_iceberg_wrong_external.py::test_undrop_iceberg_external_table[live-unqualified]
Ordinary TABLE restoration/grants/generations/context × regression -> tests/queries/test_undrop_table.py
DYNAMIC TABLE restoration × regression -> tests/queries/test_dynamic_tables.py::test_undrop_dynamic_table
DYNAMIC TABLE newest generation × regression -> tests/queries/test_dynamic_tables.py::test_undrop_dynamic_table_restores_newest_generation
DYNAMIC TABLE refresh after restoration × regression -> tests/queries/test_dynamic_tables.py::test_undropped_dynamic_table_resumes_refresh
Managed Iceberg creation and writes × regression -> tests/queries/iceberg/test_iceberg_tables.py::test_create_iceberg_table
Iceberg plain DROP × regression -> tests/queries/iceberg/test_iceberg_tables.py::test_drop_iceberg_table_via_plain_drop_table
Sibling-generation purge/ordinary UNDROP × regression -> tests/queries/iceberg/test_iceberg_tables.py::test_purge_preserves_sibling_iceberg_identity
EXTERNAL/TABLE spelling and purged-generation resolution × regression -> tests/queries/external_tables/test_drop_external_table_options.py::test_undrop_external_table_purged_generation
ICEBERG TABLE grammar/display × round trip -> vendor/sqlparser/tests/sqlparser_snowflake.rs::parse_snowflake_undrop
Polaris/REST retained data/metadata/refresh × restoration -> uncovered: env-blocked, http://localhost:8181 is worker-local and no Cloud-visible endpoint is provisioned.
Catalog-linked database × unsupported restoration -> uncovered: no constructible emulator operand; new catalog-linked database support is explicitly excluded by the spec.
* LAV-4239: Mark managed Iceberg restoration tests as S3-dependent
Add the required localstack marker to all six managed_iceberg_volume
consumers in the new restoration tests. Retain their existing assertions,
SQL, requires_s3 decorators and Cloud snapshots unchanged.
Swept both new compat files for missing backend markers; fixed all six
managed-volume consumers; existing external-volume, object-store and
external-table consumers already carry localstack markers.
Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>
## Testing
make check: passed, including vendored sqlparser tests.
.venv/bin/python scripts/guards/check_guards.py --staged: passed.
RUN_LOCALSTACK_TESTS=1 PYTEST_XDIST_ARGS='' make test-compat
PYTEST_PATH=tests/queries/iceberg/test_undrop_iceberg_table.py: 46 passed.
env -u RUN_LOCALSTACK_TESTS PYTEST_XDIST_ARGS='' make test-compat
PYTEST_PATH=tests/queries/iceberg/test_undrop_iceberg_table.py:
26 passed, 20 backend-dependent cases skipped before fixture setup.
An intermediate filtered make invocation failed because make split the
-k expression into targets; the complete-file runs above replace it.
## Test matrix
This round changes backend eligibility only, not SQL behavior. The complete
SQL matrix and Cloud-capture deviations remain recorded in 97be7beb8.
Each marked cell below passed with S3 enabled and skipped without S3.
managed lifecycle [unqualified/schema-qualified/qualified/quoted] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_managed_lifecycle
newest generation × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_newest_generation
context [other-schema/other-database/no-schema/no-database] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_context
live Iceberg collision × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_live_iceberg_collision
mixed generations [ordinary/iceberg] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_mixed_generations
view generations [view/iceberg] × backend present/absent -> tests/queries/iceberg/test_undrop_iceberg_table.py::test_undrop_iceberg_view_generations
* LAV-4239: Select Iceberg restoration tests for imported helper changes
Add the restoration test module to both helper-module selection rules.
Reproduced the importer-closure regression before the fix; all selector tests
now pass without weakening any existing assertions.
Swept imports in test_undrop_iceberg_table.py for missing importer mappings;
fixed test_iceberg_column_defaults.py and test_iceberg_tables.py selection;
conftest imports already select the entire Iceberg directory.
Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>
## Testing
.venv/bin/python -m pytest scripts/test_select_compat_tests.py -q: 228 passed.
make check: passed, including vendored sqlparser tests.
RUN_LOCALSTACK_TESTS=1 PYTEST_XDIST_ARGS='' make test-compat
PYTEST_PATH=tests/queries/iceberg/test_undrop_iceberg_table.py: 46 passed.
## Test matrix
This round changes CI selection only; no SQL behavior, tests or snapshots
changed. The SQL matrix and Cloud-capture deviations remain in 97be7beb8.
column-default helper change x importer selection -> scripts/test_select_compat_tests.py::test_changed_imported_test_module_selects_all_its_importers
table metadata helper change x importer selection -> scripts/test_select_compat_tests.py::test_changed_imported_test_module_selects_all_its_importers
selection mapping validity and regression -> scripts/test_select_compat_tests.py (228 tests)
Iceberg restoration lifecycle/error/context/dependency regression -> tests/queries/iceberg/test_undrop_iceberg_table.py (46 cases)
---------
Co-authored-by: spiral <spiral@localhost>
Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>1 parent 010ca58 commit c87b219
3 files changed
Lines changed: 8 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13262 | 13262 | | |
13263 | 13263 | | |
13264 | 13264 | | |
| 13265 | + | |
| 13266 | + | |
13265 | 13267 | | |
13266 | 13268 | | |
13267 | 13269 | | |
| |||
13314 | 13316 | | |
13315 | 13317 | | |
13316 | 13318 | | |
| 13319 | + | |
13317 | 13320 | | |
13318 | 13321 | | |
13319 | 13322 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8790 | 8790 | | |
8791 | 8791 | | |
8792 | 8792 | | |
8793 | | - | |
| 8793 | + | |
8794 | 8794 | | |
8795 | 8795 | | |
8796 | 8796 | | |
| |||
8801 | 8801 | | |
8802 | 8802 | | |
8803 | 8803 | | |
| 8804 | + | |
| 8805 | + | |
8804 | 8806 | | |
8805 | 8807 | | |
8806 | 8808 | | |
| |||
8821 | 8823 | | |
8822 | 8824 | | |
8823 | 8825 | | |
8824 | | - | |
| 8826 | + | |
8825 | 8827 | | |
8826 | 8828 | | |
8827 | 8829 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10199 | 10199 | | |
10200 | 10200 | | |
10201 | 10201 | | |
| 10202 | + | |
10202 | 10203 | | |
10203 | 10204 | | |
10204 | 10205 | | |
| |||
0 commit comments