[rshdhere]

The latest_* columns were derived from the current snapshot log, which expiring snapshots trims, so older entries started returning null. Read them from the metadata file each entry points to, only when one of those columns is projected. If that file is gone, the latest_* columns are null and a warning is logged.

Fixes #17761

[rshdhere]

PTAL @uros-b

[uros-b]

The expire-snapshots fix is the right direction: latest_* from each entry's metadata file, last snapshot-log entry rather than currentSnapshot() (so REPLACE still works), and historical IO only when a latest_* field is projected.

Two things before merge:

1. Missing-file policy. NotFoundException currently falls back to snapshotIdAsOfTime with no log, so one result mixes file-derived and time-derived rows and a missing file looks like a real no-snapshot row. An up-to-date handle cannot hit this (CatalogUtil.deleteRemovedMetadataFiles only deletes files already pruned from the log); a stale CachingCatalog handle can, after another writer committed with write.metadata.delete-after-commit.enabled. Fail-fast would break a SELECT * that works on main, so I would not go back to unguarded #17762. Prefer #16859: NotFoundException → null + LOG.warn; keep RuntimeIOException propagating. If you want to keep the time-based value, log it and say so in the description.

2. Tests. snapshotColumnsNotProjected cannot fail: deleting the files only takes the NotFoundException path, and timestamp/file do not use it. Spy the FileIO and verify(io, never()).newInputFile(previous.file()), like #16859 doesNotLoadPropertiesWhenNotProjected (overwriting with invalid JSON also works). On snapshotDetailsAfterExpiration, after.subList(0, before.size()).isEqualTo(before) already pins the expired row from values captured before expire; replace after.get(1) asserting first with second's id / schema id / sequence number on the row written while it was current.

Not asking here: Tasks.foreach / planExecutor() prefetch. That's a follow-up and should be shared with #16859 so SELECT * does not parse every retained file twice. Do not skip file reads for rows at or after the oldest retained snapshot-log timestamp — that recreates #17761 when a middle snapshot is expired.

[rshdhere]

Thanks @uros-b, both points make sense.

1. I'll switch the missing-file case to match Core, Spark, Flink: Add table properties to metadata log entries #16859: on NotFoundException, return null and LOG.warn, and let RuntimeIOException propagate. I'll drop the snapshotIdAsOfTime fallback, so a missing file no longer shows up as a time-derived row.
2. For the tests, snapshotColumnsNotProjected will spy on the FileIO and verify(io, never()).newInputFile(previous.file()), like doesNotLoadPropertiesWhenNotProjected. In snapshotDetailsAfterExpiration, I'll replace the after.get(1) check with assertions that first has second's snapshot ID, schema ID and sequence number from when it was current.

I'll leave the prefetch for a follow-up shared with #16859, and I won't skip reads at or after the oldest retained snapshot-log timestamp, since that would bring back #17761.

[rshdhere]

done with it 792b300, now let's wait for the CI to be green and ig will be ready to merge