ARSN-623: guard missing data backend client in protectAzureBlocks - #2686
Conversation
Hello darkisdude,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## development/8.5 #2686 +/- ##
===================================================
+ Coverage 74.40% 74.45% +0.04%
===================================================
Files 229 229
Lines 18546 18549 +3
Branches 3809 3810 +1
===================================================
+ Hits 13800 13810 +10
+ Misses 4741 4734 -7
Partials 5 5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4803c5b to
349dd80
Compare
A location removed from the config overlay leaves objects behind that still reference it, so this.clients[location] can be undefined. Reading a property off it threw a TypeError which, crossing an async callback that had already fired, became "Callback was already called" and was raised as an unhandledRejection - killing the cloudserver process. With client retries that took down every pod and the whole S3 endpoint. Skip block protection when no client is configured for the location: a location with no client cannot be an Azure backend. Issue: ARSN-623
349dd80 to
43a5307
Compare
SylvainSenechal
left a comment
SylvainSenechal
left a comment
There was a problem hiding this comment.
Side question : Is it normal that someone can remove a location, while their arstesca is still holding objects referencing that locations 🤔 seems like many other bug could follow from this situation 🤔
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
|
/approve |
In the queueThe changeset has received all authorizations and has been added to the The changeset will be merged in:
The following branches will NOT be impacted:
This pull request does not target the following hotfix branch(es) so they
There is no action required on your side. You will be notified here once IMPORTANT Please do not attempt to modify this pull request.
If you need this pull request to be removed from the queue, please contact a The following options are set: approve |
|
I have successfully merged the changeset of this pull request
The following branches have NOT changed:
Please check the status of the associated issue ARSN-623. Goodbye darkisdude. |
4 participants
Issue: ARSN-623
Problem
MultipleBackendGateway.protectAzureBlocks()dereferencesthis.clients[location]without checking that the client exists:The
ifguards whether the client implements the method, but never whether it exists.When a location is removed from the config overlay, objects that still reference it keep
dataStoreNamepointing at it, andcloudserver/lib/data/wrapper.jsrebuilds the clients map without it. Deleting one of those objects then throws aTypeError, which crosses anasynccallback that has already fired, becomesError: Callback was already called., is raised as anunhandledRejection— and Node exits the process.Impact observed in the field: a single
delete-objectscall took every cloudserver pod into CrashLoopBackOff and the ARTESCA S3 endpoint fully offline, because each client retry landed on and killed a fresh pod. Found during the IronMountain XDM scale campaign (OS-1095) on ARTESCA 4.3.0-rc2 /cloudserver:9.3.9with arsenal 8.4.4.This is not Azure-specific:
protectAzureBlocksruns on the shared object-delete path for every delete whenbackends.data === 'multiple'(the Zenko/XDM default). The Azure logic is inside the client, past the lookup that crashes.Change
Return early when no client is configured for the location, and log it. Skipping is correct here — the call exists only to protect Azure blocks, and a location with no configured client cannot be an Azure backend.