fix(api): make PollForMediaWikiJobsJob skip soft-deleted wikis - #1221
fix(api): make PollForMediaWikiJobsJob skip soft-deleted wikis#1221dati18 wants to merge 3 commits into
Conversation
| $this->mwHostResolver->getBackendUrlForDomain($wikiDomain) . '/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=json' | ||
| ); | ||
| } catch (UnknownWikiDomainException $e) { | ||
| Log::warning('Skipping wiki ' . $wikiDomain . ' for pending MediaWiki jobs: ' . $e->getMessage()); |
There was a problem hiding this comment.
Does this help us much? Are we just turning an error into a warning? Maybe this is still useful though.
I guess the question is what causes this to happen? Such a long runtime for this job that race conditions are common?
There was a problem hiding this comment.
Good question.
It does turn one failure mode from error to warning, but importantly it prevents a single stale domain from sabotaging the whole polling run.
To answer "Why this happens":
- The job reads a list of domains.
- Later, for each domain, resolver does another lookup.
- If a wiki is soft-deleted (or otherwise missing) between those moments, resolver throws
UnknownWikiDomainException.
The root cause I think is "double-read on mutable data" under multiple updates. The longer the runtime, the longer the gap between 1 and 3, the higher the probability. So basically it's the gap between the time of "reading the data" vs "using the data"
There was a problem hiding this comment.
nice! Sounds like we are getting closer to the root cause.
So could we refactor the job so we do this look up earlier? Or even in a single query?
I guess there might still be a case where the wiki really has now gone away
There was a problem hiding this comment.
Good idea, so you want to weed out bad domains instead of just "skipping" it?
There was a problem hiding this comment.
I think you are correct that there is a double read. Actually there is a triple read:
- once to get the list of Wikis
- once to get the backendHost
- once when then MW backendHost determines the configuration of the Wiki
I am suggesting we merge the first two cases into one and get the correct backend hosts at the same time as we get the list of Wikis. There is still a chance that by the time we get to polling the Wikis that the 3rd read fails because it has been deleted but this is still a better solution.
Are you sure that marking the job as failed is sabotaging the whole run? I assumed that the job would still iterate over all Wikis but it would retry (sooner than planned) because it was marked as failed.
2 participants
Bug: T433575