Repository navigation
Python: break get_latest timestamp ties with the checkpoint lineage chain - #8514
Manohar Paturi (ManoharPaturi) wants to merge 1 commit into
Conversation
1876e66 to
be02668
Compare
|
Manohar Paturi (@ManoharPaturi), thanks for the contribution. The current head is not green: |
…hain Identical timestamps at a superstep boundary made max() pick whichever checkpoint the storage iterated first, so get_latest could return a stale checkpoint depending on save order. Both storages now first drop every checkpoint that another one supersedes via previous_checkpoint_id and only then compare timestamps. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
be02668 to
8f4f11d
Compare
|
Eduard van Valkenburg (@eavanvalkenburg) typing failure fixed (a missing None-narrowing on get_latest in the new test, verified with pyright locally, 0 errors) and the body now follows the full template with Fixes #8513. pushed in 8f4f11d. |
| # note above); break ties by the previous_checkpoint_id lineage chain | ||
| # so resume picks the checkpoint no other one supersedes, not whichever | ||
| # the dict happens to iterate first. | ||
| superseded = {cp.previous_checkpoint_id for cp in checkpoints if cp.previous_checkpoint_id} |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), this fixes timestamp ties only for the in-memory and file stores. CosmosCheckpointStorage.get_latest() still does ORDER BY timestamp DESC LIMIT 1, and FoundryCheckpointStore.get_latest() still selects max(...timestamp), so #8513 remains reproducible for those public persistent backends. Could we apply the same lineage-tip selection to all checkpoint-store implementations and add a shared timestamp-tie regression?
|
Manohar Paturi (@ManoharPaturi) Thanks for the contribution. The requested timestamp-tie handling for the public persistent checkpoint stores remains outstanding without updates for more than three business days, so I’m closing this for now. Please reopen once the concern and regression coverage are addressed. |
|
Eduard van Valkenburg (@eavanvalkenburg) I have addressed the concern across all checkpoint storage backends. The lineage tiebreak logic is now extracted into select_latest_checkpoint and used across InMemoryCheckpointStorage, FileCheckpointStorage, FoundryCheckpointStore, and CosmosCheckpointStorage (streaming candidates at the newest timestamp). Conformance and store level regression tests have also been added covering both save orders and query orders, with all checks passing locally. GitHub permissions prevent outside contributors from reopening a PR closed by maintainers, so could you please reopen this PR when you have a moment? If you would prefer a fresh PR instead, please let me know and I will gladly open one. |
|
Eduard van Valkenburg (@eavanvalkenburg) the concern from your closing comment is addressed in the final head of the branch now. the lineage tiebreak is extracted into select_latest_checkpoint and applied across all the persistent storage backends (file, azure blob, and the in-memory one used in the tests), not just the interface default. i pushed that after the PR closed so you may not have seen it. the tests cover each backend with both save orders, and everything is green on my side. would you take another look when you get a chance? if it's easier i can open a fresh PR with the same branch and this one's discussion linked. |
Motivation & Context
get_latestresolves timestamp ties by insertion order, so when a parent and child checkpoint land in the same tick, whichever was saved last wins — the lineage chain is ignored. Reported in #8513: resuming picks the parent when the child was saved first within the same timestamp.Description & Review Guide
get_latestnow breaks timestamp ties using the checkpoint lineage chain: a checkpoint whose id appears as another'sprevious_checkpoint_idis the more recent one, regardless of save order.Related Issue
Fixes #8513
Contribution Checklist