[moomindani]

Summary

Closes #15593

This revives #15594, which the stale bot closed on 2026-05-21 after ten weeks without a single review. GitHub does not offer a reopen on that PR, so this is a fresh PR from the same branch, rebased onto current main. Spark 3.4 support was removed upstream in #14122, so the v3.4 copies are dropped — this now touches Spark 3.5, 4.0 and 4.1 (6 files instead of 8). Nothing else changed from the closed PR.

Introduces a new scd_type2 => boolean parameter to the create_changelog_view Spark procedure that natively produces a Slowly Changing Dimensions Type-2 (SCD Type-2) history view from an Iceberg changelog table, without requiring users to write custom SQL.

New columns added to the view

Column	Type	Nullable	Description
_valid_from	TIMESTAMP	no	Commit timestamp when this version became active
_valid_to	TIMESTAMP	yes	Commit timestamp when superseded; NULL if still active
_is_current	BOOLEAN	no	true when _valid_to IS NULL AND _change_type != 'DELETE'

Example

CALL catalog.system.create_changelog_view(
    table              => 'db.products',
    changelog_view     => 'products_history',
    identifier_columns => array('product_id'),
    scd_type2          => true
)

Given inserts and updates to products, the resulting view:

product_id	name	_change_type	_valid_from	_valid_to	_is_current
1	Widget	INSERT	2024-01-01	2024-02-01	false
1	Widget Pro	UPDATE_AFTER	2024-02-01	NULL	true

Demo output (from actual Spark 4.1 test run)

Setup:

CREATE TABLE products (id INT NOT NULL, data STRING) USING iceberg;
ALTER TABLE products SET IDENTIFIER FIELDS id;
ALTER TABLE products ADD PARTITION FIELD id;

INSERT INTO products VALUES (1, 'Widget');
INSERT INTO products VALUES (2, 'Gadget');
INSERT OVERWRITE products VALUES (1, 'Widget Pro');   -- update entity 1

Without scd_type2 (existing behavior):

CALL catalog.system.create_changelog_view(
    table              => 'db.products',
    identifier_columns => array('id')
)

id   data         _change_type    _change_ordinal   _commit_snapshot_id
---------------------------------------------------------------------------
1    Widget       INSERT          0                 5325177678658542000
1    Widget Pro   UPDATE_AFTER    1                 263949438103749135
1    Widget       UPDATE_BEFORE   1                 263949438103749135
2    Gadget       INSERT          1                 1299271826648018743

Users must manually write window functions to derive validity intervals.

With scd_type2 => true (this PR):

CALL catalog.system.create_changelog_view(
    table              => 'db.products',
    identifier_columns => array('id'),
    scd_type2          => true
)

id   data         _change_type   _change_ordinal   _commit_snapshot_id    _valid_from               _valid_to                 _is_current
-------------------------------------------------------------------------------------------------------------------------------------------------
1    Widget       INSERT         0                 5325177678658542000    2026-04-13 11:23:44.977   2026-04-13 11:23:45.543   false
1    Widget Pro   UPDATE_AFTER   2                 263949438103749135     2026-04-13 11:23:45.543   null                      true
2    Gadget       INSERT         1                 1299271826648018743    2026-04-13 11:23:45.269   null                      true

Key differences:

• UPDATE_BEFORE rows filtered out (intermediate artifacts not meaningful in SCD Type-2)
• _valid_from / _valid_to computed automatically via LEAD() window over snapshot timestamps
• _is_current convenience flag: true only when _valid_to IS NULL AND _change_type != 'DELETE'

Querying current state:

SELECT id, data FROM products_changelog WHERE _is_current = true ORDER BY id

id   data
------------------
1    Widget Pro
2    Gadget

No custom window logic needed.

Design choices

• NULL sentinel for open-ended rows (not 9999-12-31) — aligns with dbt and Databricks DLT
• _valid_from/_valid_to naming — aligns with dbt dbt_valid_from/dbt_valid_to and Kimball semantics
• UPDATE_BEFORE rows filtered out — intermediate artifacts not meaningful in an SCD Type-2 view
• DELETE rows kept with _is_current = false — tracks hard deletes
• _is_current convenience flag added — avoids the two-condition footgun (_valid_to IS NULL AND _change_type != 'DELETE')

Constraints

• Requires identifier_columns (explicit or from table schema)
• Incompatible with net_changes = true
• Implicitly forces compute_updates = true

Note on snapshot expiration

The scd_type2 mode inherits the same snapshot expiration behavior as the existing create_changelog_view. If snapshots have been expired, the changelog scan itself omits the corresponding rows — the SCD Type-2 layer does not introduce any additional data loss or new failure modes. Specifically:

• Expired snapshot, data files still present: changelog scan silently omits rows from the expired snapshot. Both base and scd_type2 modes return the same reduced set of rows. _valid_from/_valid_to/_is_current are correct relative to the rows the changelog scan returns.
• Expired snapshot, data files garbage-collected: changelog scan fails with FileNotFoundException. Both base and scd_type2 modes fail identically.

Changes

• CreateChangelogViewProcedure.java — Spark v3.5, v4.0, v4.1
• TestCreateChangelogViewProcedure.java — Spark v3.5, v4.0, v4.1

Test plan

• testScdType2BasicInsertUpdateDelete — INSERT → UPDATE → DELETE lifecycle, verifies all 3 new columns
• testScdType2CurrentRows — only non-deleted, most-recent rows have _is_current = true
• testScdType2HardDelete — DELETE row has _valid_to = NULL, _is_current = false
• testScdType2RequiresIdentifierColumns — error when no identifier columns
• testScdType2IncompatibleWithNetChanges — error when combined with net_changes = true
• testScdType2OutputSchema — verifies all 8 output columns in correct order

[moomindani]

@huaxingao @szehon-ho would one of you be able to take a look?

This revives #15594, which the stale bot closed in May after ten weeks without a review; GitHub offers no reopen on that one, so this is a fresh PR from the same branch. It is rebased onto current main, the spark/v3.4 copies are dropped since #14122 removed Spark 3.4 support, and all eight spark-tests jobs are green (3.5 × 2.12/2.13, 4.0, 4.1, core and extensions).

The parts that most need a reviewer's judgment are the SCD Type-2 semantics rather than the plumbing:

• NULL as the open-ended sentinel for _valid_to, rather than 9999-12-31
• _valid_from / _valid_to naming, following dbt and Kimball rather than inventing Iceberg-specific names
• dropping UPDATE_BEFORE rows, which are intermediate artifacts in an SCD Type-2 view
• keeping DELETE rows with _is_current = false, so hard deletes stay visible

If any of those should go the other way, I am happy to change them. The full rationale and a demo output are in the PR description.

[szehon-ho]

Hi @moomindani , interesting. Still thinking but not entirely sure yet of the approach.

Iceberg tables typically have quite a short history, snapshots are usually expired quickly because of compliance, data storage.. the value of this will be quite short.

To me it make sense to track the Iceberg values in a separate SCD type 2 table for longer history?

[moomindani]

Thanks @szehon-ho — that premise is right, and I think it is worth naming the options explicitly, because "a separate SCD Type-2 table" can be reached in more than one way and I am not sure which one you have in mind.

On the horizon: the changelog scan can only see changes whose snapshots are still around, so a view built directly on it covers exactly the retention window and no more. How long that window is, though, is a choice rather than a fixed limit — history.expire.max-snapshot-age-ms defaults to 5 days but can be set arbitrarily high, min-snapshots-to-keep also holds snapshots back, and nothing expires at all until expire_snapshots actually runs, so the effective horizon is the property plus the maintenance schedule. A table that keeps months of snapshots would give months of SCD Type-2 view.

That said, I would not build a feature on the assumption that the window is long. Retention gets chosen for storage cost, metadata size and compliance deletion, not for a downstream SCD Type-2 use case, so on most tables it will be short — which is your point, and I agree with where it lands.

As far as I can see there are three ways to end up with a long-lived SCD Type-2 table:

1. Dual write — the writer appends to the main table and to an SCD Type-2 table in the same job. No dependency on retention, but it changes every write path, and Iceberg has no cross-table atomic commit today (there was a dev@ thread on multi-table transactions in July), so the two tables can diverge when one commit fails.
2. SCD Type-2 as the base table — store only versions and derive the current cross-section on read. Nothing is ever lost, but every consumer then pays for the _is_current / latest-per-key filter, and existing readers of the table have to change.
3. Derive from the changelog and store it — the write path and the consumers stay as they are, and a scheduled job appends the new versions to a table you own. The cost is operational: that job has to run more often than snapshots are expired on the source table, and history missed in that window cannot be recovered.

This PR is the derivation step of option 3. It stores nothing itself; it turns changelog rows into version intervals — joining snapshot timestamps, applying the window function, dropping the UPDATE_BEFORE images, deciding what a delete means — which is the part users otherwise hand-write in SQL for every batch, and the reason this came up in the first place.

Since the option needed documentation anyway, I wrote that up rather than just asserting it: ef48e01a1 adds a "Building a Long-Lived SCD Type-2 Table" section that states the retention bound and shows the loop — run with options => map('start-snapshot-id', ...), MERGE to close the version that was open in the target table, append the new ones, and record the end snapshot for the next run.

Which of the three did you have in mind? If it is 1 or 2, that is a different feature from this one and I would not claim this PR covers it. If it is 3, this is the piece that makes it a few lines of SQL instead of a hand-rolled window query.

[moomindani]

@szehon-ho a nudge on this one — the question at the end of my 26 August comment is the only thing holding it, and I would rather have an answer either way than leave it sitting.

To make it cheap to answer: the three routes to a long-lived SCD Type-2 table were (1) dual write, (2) SCD Type-2 as the base table, (3) derive from the changelog and store it. This PR is only the derivation step of (3) — it stores nothing itself. If you meant (1) or (2), those are different features and I do not claim this covers them.

One correction: that comment pointed at ef48e01a1 for the documentation, but I rebased the branch on 30 August and that SHA no longer exists. The docs are now in 40f83f48c, under "Building a Long-Lived SCD Type-2 Table" in docs/docs/spark-procedures.md; it states the retention bound plainly and shows the incremental loop.

And if your conclusion is that the retention window makes this not worth carrying inside the procedure, please say so and I will close the PR. Rebased onto current main, CI green (38/38).

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[moomindani]

Not stale, this is waiting on one design question. @szehon-ho, did "a separate SCD Type-2 table" in your Aug 25 review mean deriving it from the changelog and storing the result (route 3 in my Aug 26 comment, which is the step this PR provides), or a different approach? The branch is mergeable and CI is green.