Repository navigation
CTM-678: fix Galaxy disk stuck Ready after createInstance fails before VM exists - #4938
Merged
Merged
Conversation
… VM exists When createInstance fails immediately (e.g. ZONE_RESOURCE_POOL_EXHAUSTED), Leo has already created the NFS disk in GCP but the VM was never provisioned. GCP returns null for getLastDetachTimestamp on a disk that was never attached. The disk detach wait compared Some(null) == Some(null) → done=false, looped for 250 s (50 × 5 s), timed out, and the exception was swallowed by handleErrorWith in cleanUpAfterCreateAppError. The disk stayed Ready in the DB. On the next create attempt Leo found the Ready disk, set restore=true, and ansible-pull aborted because the disk has no Galaxy data. Cleanup in restore mode does not delete the disk (diskId = None), so the disk is stuck permanently. Fix: normalise null → None via Option(ts) when recording the original detach timestamps for both the NFS and postgres disks. In DoneCheckable, treat a None original as 'never attached → safe to delete immediately', skipping the detach wait entirely for these disks. Normal deletion (disk was previously attached and has a non-null timestamp) is unaffected. Ticket: https://broadworkbench.atlassian.net/browse/CTM-678 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LizBaldo
force-pushed
the
fix/galaxy-disk-detach-never-attached
branch
from
October 5, 2026 20:25
c3f60de to
f6658b9
Compare
lucymcnatt
approved these changes
Oct 5, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #4938 +/- ##
========================================
Coverage 73.71% 73.71%
========================================
Files 131 131
Lines 11262 11263 +1
Branches 916 933 +17
========================================
+ Hits 8302 8303 +1
Misses 2960 2960
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
createInstancefails immediately (e.g.ZONE_RESOURCE_POOL_EXHAUSTED), Leo creates the NFS disk in GCP but the VM is never created. The disk is created but never attached to any VM.The disk cleanup in
deleteApprecords the disk'sgetLastDetachTimestampand then polls until the timestamp changes (indicating detach). For a never-attached disk GCP returnsnullfor that field. Both the original and every subsequent reading areSome(null), so they compare equal — theDoneCheckablenever fires and the wait times out after 250 s (50 × 5 s). The timeout exception is swallowed byhandleErrorWithincleanUpAfterCreateAppError, and the disk staysReadyin the DB.On the next create attempt Leo finds the
Readydisk, setsrestore=true, passesrestore_galaxy=trueto ansible, and ansible aborts because the disk has no Galaxy data. Cleanup in restore mode does not delete the disk (diskId = None), so the disk is stuck permanently.This was exposed more prominently by the fast-fail change in #4933: before that,
createInstanceappeared async and Leo polled for 30 min, giving time for the VM (and disk attachment) to reach a terminal state before cleanup ran.Fix
null → NoneviaOption(getLastDetachTimestamp)when recording original timestamps for both the NFS data disk and the postgres disk.diskDetachDone: if the original timestamp isNone(disk was never attached), returntrueimmediately — no wait needed, the disk is already safe to delete.Ticket: https://broadworkbench.atlassian.net/browse/CTM-678
🤖 Generated with Claude Code