Partial-progress compaction can lose manifest-list encryption keys #17989

Activity

1. smaheshwar-pltr

   added

   bugSomething isn't workingSomething isn't working

   on Sep 7, 2026
2. smaheshwar-pltr commented on Sep 7, 2026

   

   smaheshwar-pltr

   on Sep 7, 2026

   ContributorAuthor

   More actions

   Example reproduction below. It's a bit heavy; there may have been an easier repro here but couldn't find one easily.

   @TestTemplate
   public void testConcurrentCompactionPreservesEncryptionKeys() {
       // Partitioned encrypted table: one file per partition => many file groups for rewrite
       sql("DROP TABLE %s", tableName);
       sql(
           "CREATE TABLE %s (id bigint, data string) USING iceberg PARTITIONED BY (id) "
               + "TBLPROPERTIES ('encryption.key-id'='%s', 'format-version'='3')",
           tableName, UnitestKMS.MASTER_KEY_NAME1);
       sql("INSERT INTO %s SELECT id, 'v' FROM (SELECT explode(sequence(0, 511)) AS id)", tableName);
   
       validationCatalog.initialize(catalogName, catalogConfig);
       Table table = validationCatalog.loadTable(tableIdent);
   
       SparkActions.get(spark)
           .rewriteDataFiles(table)
           .binPack()
           .option("partial-progress.enabled", "true")
           .option("partial-progress.max-commits", "2000")
           .option("max-concurrent-file-group-rewrites", "128")
           .option("rewrite-all", "true")
           .execute();
   
       assertThat(currentDataFiles(validationCatalog.loadTable(tableIdent))).isNotEmpty();
     }
3. smaheshwar-pltr commented on Sep 7, 2026

   

   smaheshwar-pltr

   on Sep 7, 2026

   Last edited by smaheshwar-pltr

   ContributorAuthor

   More actions

   A couple notes: (went down a few routes investigating this, so wanted to track)

   • The root cause is that of Concurrent commits through one Table can lose manifest-list encryption keys #17990 - each rewrite worker writes with the encryption manager it captured, while a concurrent Hive refresh can "install" a different manager before metadata commit. The successful snapshot's manifest-list key is then present only in that original, "lost" manager, and so not committed.
   • Separately, I think that it's odd that partial progress runs commits concurrently, leading to excessive and unnecessary self-conflicts, retries, backoff etc. I have Core: Write rewrite manifests concurrently but commit one at a time #17986 for this which shows significant improvements when benchmarked.
4. smaheshwar-pltr commented on Sep 24, 2026

   

   smaheshwar-pltr

   on Sep 24, 2026

   ContributorAuthor

   More actions

   Fixed by #17984
5. smaheshwar-pltr

   closed this as completedon Sep 24, 2026