Skip to content

CTM-678: fix Galaxy disk stuck Ready after createInstance fails before VM exists - #4938

Merged
LizBaldo merged 2 commits into
developfrom
fix/galaxy-disk-detach-never-attached
Oct 6, 2026
Merged

LizBaldo merged 2 commits into
developfrom
fix/galaxy-disk-detach-never-attached

Conversation

@LizBaldo

@LizBaldo LizBaldo commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

Problem

When createInstance fails immediately (e.g. ZONE_RESOURCE_POOL_EXHAUSTED), Leo creates the NFS disk in GCP but the VM is never created. The disk is created but never attached to any VM.

The disk cleanup in deleteApp records the disk's getLastDetachTimestamp and then polls until the timestamp changes (indicating detach). For a never-attached disk GCP returns null for that field. Both the original and every subsequent reading are Some(null), so they compare equal — the DoneCheckable never fires and the wait times out after 250 s (50 × 5 s). The timeout exception is swallowed by handleErrorWith in cleanUpAfterCreateAppError, and the disk stays Ready in the DB.

On the next create attempt Leo finds the Ready disk, sets restore=true, passes restore_galaxy=true to ansible, and ansible aborts because the disk has no Galaxy data. Cleanup in restore mode does not delete the disk (diskId = None), so the disk is stuck permanently.

This was exposed more prominently by the fast-fail change in #4933: before that, createInstance appeared async and Leo polled for 30 min, giving time for the VM (and disk attachment) to reach a terminal state before cleanup ran.

Fix

  • Normalise null → None via Option(getLastDetachTimestamp) when recording original timestamps for both the NFS data disk and the postgres disk.
  • Add an early-exit clause to diskDetachDone: if the original timestamp is None (disk was never attached), return true immediately — no wait needed, the disk is already safe to delete.

Ticket: https://broadworkbench.atlassian.net/browse/CTM-678

🤖 Generated with Claude Code

… VM exists

When createInstance fails immediately (e.g. ZONE_RESOURCE_POOL_EXHAUSTED),
Leo has already created the NFS disk in GCP but the VM was never provisioned.
GCP returns null for getLastDetachTimestamp on a disk that was never attached.

The disk detach wait compared Some(null) == Some(null) → done=false, looped
for 250 s (50 × 5 s), timed out, and the exception was swallowed by
handleErrorWith in cleanUpAfterCreateAppError. The disk stayed Ready in the DB.

On the next create attempt Leo found the Ready disk, set restore=true, and
ansible-pull aborted because the disk has no Galaxy data. Cleanup in restore
mode does not delete the disk (diskId = None), so the disk is stuck permanently.

Fix: normalise null → None via Option(ts) when recording the original detach
timestamps for both the NFS and postgres disks. In DoneCheckable, treat a None
original as 'never attached → safe to delete immediately', skipping the detach
wait entirely for these disks. Normal deletion (disk was previously attached
and has a non-null timestamp) is unaffected.

Ticket: https://broadworkbench.atlassian.net/browse/CTM-678

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@LizBaldo
LizBaldo force-pushed the fix/galaxy-disk-detach-never-attached branch from c3f60de to f6658b9 Compare October 5, 2026 20:25

@jgainerdewar jgainerdewar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh

@codecov

codecov Bot commented Oct 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.71%. Comparing base (b98d3d4) to head (063b69a).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff            @@
##           develop    #4938   +/-   ##
========================================
  Coverage    73.71%   73.71%           
========================================
  Files          131      131           
  Lines        11262    11263    +1     
  Branches       916      933   +17     
========================================
+ Hits          8302     8303    +1     
  Misses        2960     2960           
Files with missing lines Coverage Δ
.../leonardo/monitor/LeoPubsubMessageSubscriber.scala 76.83% <100.00%> (+0.02%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b98d3d4...063b69a. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@LizBaldo
LizBaldo merged commit b8710d1 into develop Oct 6, 2026
22 of 23 checks passed
@LizBaldo
LizBaldo deleted the fix/galaxy-disk-detach-never-attached branch October 6, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants