Repository navigation
Fix: Sometimes manage_queues does not dump all available messages (#3663) - #8414
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8414 +/- ##
===========================================
+ Coverage 85.00% 85.42% +0.41%
===========================================
Files 170 171 +1
Lines 26085 26423 +338
===========================================
+ Hits 22174 22571 +397
+ Misses 3911 3852 -59 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
hannes-ucsc
force-pushed
the
issues/hannes-ucsc/3663-manage-queues
branch
from
October 22, 2026 00:40
396f7c4 to
208aa31
Compare
Member
Author
Security design review
|
The entity types in an AnVIL bundle are the names of the BigQuery tables the entities are drawn from, and therefore carry an `anvil_` prefix. The set of types to redact lists them without that prefix, so the membership test never holds, and `--redaction-key` quietly writes the bundle out unredacted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bundle holds its orphans separately from its entities, and only the latter were redacted, leaving an orphaned biosample, diagnosis or donor in the clear. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SQS redrive facility moves messages server-side, so unlike the dump and feed commands it is not subject to the limitations of receiving from a FIFO queue, where the messages of a group behind an in-flight head remain unreachable. The source queue must be configured as the dead-letter queue of another queue, and both queues must be of the same type. The command waits for the move to complete. Since SQS only reports the handle of a task while that task is running, a finished task is identified by when it started. Also make the optional arguments of the client method keyword-only, matching how every call site already passes them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check reads the queue property of the dump envelope, which a later commit replaces. Validation returns at the end of this branch, in terms of the type of message rather than the identity of the queue, so that feeding the contents of a fail queue into the corresponding work queue no longer requires an override. The --force flag goes with the check, and the procedure in OPERATOR.rst no longer refers to force-feeding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_dump_messages condensed the messages it was given, but the cleanup of an interrupted feed passes it the already condensed messages it read from the dump file. The resulting AttributeError was raised while handling the exception that interrupted the feed, hiding the actual cause of the interruption from the operator. The messages are now condensed by _dump, the only caller that holds uncondensed ones. Present since 2cf70b9. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cleanup of an interrupted feed rewrote the dump file from the complete list of messages instead of the ones that had not been sent yet, so feeding the file again sent those messages a second time. The list of remaining messages was computed but only used to decide whether any message had been sent at all. Present since 2cf70b9. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither command was covered by a test. One test round-trips the messages of a queue through a dump file, the other interrupts a feed and covers the two defects fixed in dcf0d854f and eea840207, by asserting that the exception which interrupted the feed is the one that propagates, and that the file is left holding exactly the messages that were not sent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Appending to a dump file is a prerequisite for deleting each batch of messages as it is written, which is in turn what allows a FIFO queue to be dumped completely. This commit only changes the format, leaving the dump to collect every message in memory before writing it, and the feed to read the whole file into memory. The first line of a dump is the envelope, each subsequent line is one message. Dumps written by earlier versions are no longer accepted, and dump_all now names its files after the queue with a .jsonl suffix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A message held in flight blocks the rest of its group, so a FIFO queue cannot be dumped completely unless each batch is deleted from the queue before the next one is received. Each batch is now written to the dump file and flushed to disk before it is deleted, so that an interrupted dump leaves behind every message it removed from the queue. This also keeps the number of in-flight messages at one batch, well below the limit SQS imposes on a queue, and it bounds the time a message is invisible by the duration of one batch instead of the whole dump. Because the messages of a dump that does not delete them must be held in flight until the dump ends, or receiving them again would make the dump run forever, such a dump still captures only the messages at the head of each group. It now says so before it starts. The dump file is no longer written through write_file_atomically, whose removal of the temporary file on failure would discard exactly those messages that were already deleted from the queue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A short poll samples a subset of the SQS hosts and can therefore return nothing while messages remain, ending a dump prematurely. This affects FIFO and standard queues alike. A long poll queries all hosts, and the documented maximum wait is used because AWS notes that lower values make empty responses more likely. The wait is an attribute so that tests can disable it. Moto honours the parameter by blocking, and every drain ends on an empty receive, which would otherwise add twenty seconds to each one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wait after purging a queue added the number of available messages twice and never counted the ones in flight, so it could return while messages were still being processed, just before the readers of the queue are enabled again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An empty receive ended a dump even when the queue still reported messages. The dump now consults those counts, discounting the messages it holds in flight itself, and keeps polling while they disagree with the empty receive. The counts are approximate, and the producers of a queue keep adding to it unless they are disabled, so polling is bounded: after three consecutive empty receives the dump gives up and says what the queue still reports. The names of the three counting attributes are now defined once, and the wait after purging a queue uses them, too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The --no-json-body flag stored the body of a message verbatim instead of embedding the JSON it contains. Nothing but the command line ever set it, and every message azul queues carries a JSON body, so a body that the flag was needed for could only come from outside azul. The inverse operation loses the branch that re-serialized such a body, which means a dump written with the flag can no longer be fed back. Dumps written before this branch could not be, either, since the format has changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The class is not slotted because its cached property caches by writing to the instance dictionary, which a slotted class lacks. The sole field keeps its leading underscore, so that attrs names the constructor argument after it without one, as the callers already do. Only the constant naming the attributes that count the messages in a queue is annotated, so it is the only one attrs would have mistaken for a field, and it is declared as a class variable to prevent that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two descriptions differed only in whether they spoke of one queue and file or of several, which the wording now avoids. One of them also said "the its file". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two subcommands defined the option identically, which left them free to drift apart. They now inherit it from a parser that carries nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lambdas reading the work queues produce the messages in every queue, so a dump taken while they are running is likely to miss messages. The dump now consults the work queues, and if they hold messages, insists on a decision: --safe disables those lambdas for its duration, --unsafe accepts a dump that is probably incomplete. Combining --unsafe with --delete is refused, because it would discard the messages the dump missed. A dump taken while the work queues are empty needs no decision, and only warns that nothing kept messages from arriving while it ran. The two flags write to one tri-state attribute, so they cannot both be in effect. The mutually exclusive group they share only improves the error message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An interrupted feed rewrote the dump with the messages it had not submitted, which both destroyed the dump it was fed from and left nothing to say how far it had come. It now records that in a file beside the dump, rewritten after every batch, and skips as many messages when it is run again. The file names the queue it fed, so that a dump interrupted on its way into one queue is not silently truncated on its way into another. The --delete flag now removes the dump once every message in it was submitted, instead of removing them from it one batch at a time. A crash between submitting a batch and recording it resubmits that batch when the feed is resumed. On a FIFO queue, SQS discards the resulting duplicates, because the deduplication IDs are carried over from the dump. On a standard queue it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dump now records the type of message it holds, and a feed refuses a dump whose type is not the one the target queue is expected to hold. There are three mutually incompatible types, each named after the class whose instances are the bodies of those messages. Those classes are defined in modules that import this one, hence the deferred import. Unlike the check this replaces, which compared the queue a dump was taken from against the one it is fed into, this one permits feeding the contents of a fail queue into the work queue it belongs to, which is what the operator manual prescribes. It has no override, because no queue can do anything with a message of another type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dump records the queue it was taken from, and a feed refuses a dump whose deployment is not the current one, unless --foreign says to proceed. Type compatibility does not catch this, because the messages of one deployment are perfectly well-formed messages of another. Moving messages between deployments is a legitimate thing to do, for instance to reproduce a failure in a personal deployment, which is why this check has an override while the one on the type of message does not. The queue is recorded by name rather than by URL, because the name is what carries the deployment, and the URL adds an account and a region to a file that operators read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dump used to replace the contents of the file it was given, silently discarding an earlier dump. It now refuses that file unless --overwrite says to replace it, or --append says to add to it. Appending compares the header of the file against the one the dump would write, so that a dump of one queue is not appended to a dump of another. The two flags write to one tri-state attribute, as the flags deciding what a dump does about the queue worker lambdas do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every condition the queue management refuses to work under is a demand it makes of its input or of the state of the deployment, not a defect in itself, which is what R is for. Two of them were stated as branches of a chain that had no condition to assert, and are now stated as the conditions under which a dump of a busy deployment, and a dump to a file that exists, may proceed. The chain deciding whether to append collapses as a result, the attribute being a plain boolean once it is known not to be None. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The procedure spoke of notifications and had the operator repeat it for tallies afterwards, as an aside. It now covers both at once, in numbered steps, and prescribes feeding the tallies first, they being downstream from the notifications. Between dumping and feeding, the operator is now asked to triage the failures and to agree on a remedy, rather than to feed the messages back as they are. Messages are now fed into the work queue, with the retry queue reserved for the case where most of them are known to need the memory and the timeout of the retry lambda, instead of being fed into the retry queue to save time. The constraints the script enforces on either step are described as well, along with the flags that relax those that may be relaxed. The instructions for tracking the remaining failures in tickets are removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Indexing a bare dict yields Any, which spreads to the body and the URL of an existing PR. Typing the response as JSON makes that indexing yield a JSON value, which json_str narrows to the string both of them are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hannes-ucsc
force-pushed
the
issues/hannes-ucsc/3663-manage-queues
branch
from
October 22, 2026 06:05
208aa31 to
9c31eed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issues: #3663
Checklist
Author
A01PR is assigned to the authorA02Status of PR is In progressA03PR is a draftA04Target branch isdevelopA05Name of PR branch matchesissues/<GitHub handle of author>/<issue#>-<slug>A06PR is linked to all issues it (partially) resolvesA07Status of linked issues is In progressA08PR description links to linked issuesA09PR title matches1 that of a linked issue or comment in PR explains why they're differentA10PR title references all linked issuesA11For each linked issue, there is at least one commit whose title references that issue1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
B01Addedptag to titles of partial commitsB02This PR is labeledpartialor completely resolves all linked issuesB03This PR partially resolves each of the linked issues or does not have thepartiallabelAuthor (reindex)
C01Addedrtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentC02This PR is labeledreindex:devor the changes introduced by it will not require reindexing ofdevC03This PR is labeledreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevC04This PR is labeledreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodC05This PR is labeledreindex:prodor the changes introduced by it will not require reindexing ofprodC06This PR is labeledreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAuthor (mirror)
D01This PR is labeledmirror:devor the changes introduced by it will not require mirroring ofdevD02This PR is labeledmirror:anvildevor the changes introduced by it will not require mirroring ofanvildevD03This PR is labeledmirror:anvilprodor the changes introduced by it will not require mirroring ofanvilprodD04This PR is labeledmirror:prodor the changes introduced by it will not require mirroring ofprodD05This PR is labeledmirror:partialand its description documents the specific mirroring procedure fordev,anvildev,anvilprodandprodor requires a full mirroring or carries none of the labelsmirror:dev,mirror:anvildev,mirror:anvilprodandmirror:prodAuthor (API changes)
E01This PR and its linked issues are labeledAPIor this PR does not modify a REST APIE02Addeda(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIE03Updated REST API version number inapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
F01Ranmake docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableF02Documented upgrading of deployments in UPGRADING.rst or this PR does not require upgrading deploymentsF03Addedutag to commit title or this PR does not require upgrading deploymentsF04This PR is labeledupgradeor does not require upgrading deploymentsF05This PR is labeleddeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasonF06This PR is labeleddeploy:gitlabor does not require deploying thegitlabcomponentF07This PR is labeleddeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
G01AddedFtag to main commit title or this PR does not include permanent fix for a temporary hotfixG02Reverted the temporary hotfixes for any linked issues or the none of the stable branches (anvilprodandprod) have temporary hotfixes for any of the issues linked to this PRAuthor (before every review)
H01Rebased PR branch ondevelop, squashed fixups from prior reviewsH02Ranmake requirements_updateor this PR does not modifypyproject.tomlH03AddedRtag to commit title or this PR does not modifyuv.lockH04This PR is labeledreqsor does not modifyuv.lockH05make integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomeH06PR is awaiting requested review from a peerH07Status of PR is Review requestedH08PR is assigned to only the peer and the authorPeer reviewer (after approval)
Note that after requesting changes, the PR must be assigned to only the author.
J01Actually approved the PRJ02PR is not a draftJ03PR is awaiting requested review from system administratorJ04Status of PR is Review requestedJ05PR is assigned to only the system administrator and the authorSystem administrator (after approval)
K01Actually approved the PRK02Labeled linked issues asdemoorno demoK03Commented on linked issues about demo expectations or all linked issues are labeledno demoK04Decided if PR can be labeledno sandboxK05A comment to this PR details the completed security design reviewK06PR title is appropriate as title of merge commitK07N reviewslabel is accurateK08Status of PR is ApprovedK09PR is assigned to only the operator and the authorOperator
L01Checkedreindex:…labels andrcommit title tagL02Checkedmirror:…labelsL03Checked that demo expectations are clear or all linked issues are labeledno demoL04Squashed PR branch and rebased ontodevelopL05Sanity-checked historyL06Pushed PR branch to GitHubOperator (deploy
.sharedand.gitlabcomponents)M01Ran_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:sharedM02Ran_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) or this PR is not labeleddeploy:gitlabM03Ran_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:sharedM04Ran_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) or this PR is not labeleddeploy:gitlabM05Checked the items in the next section or this PR is labeleddeploy:gitlabM06PR is assigned to only the system administrator and the author or this PR is not labeleddeploy:gitlabSystem administrator (post-deploy of
.gitlabcomponent)N01Background migrations fordev.gitlabare complete or this PR is not labeleddeploy:gitlabN02Background migrations foranvildev.gitlabare complete or this PR is not labeleddeploy:gitlabN03PR is assigned to only the operator and the authorOperator (deploy runner image)
P01Ran_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnerP02Ran_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnerOperator (sandbox build)
Q01Addedsandboxlabel or PR is labeledno sandboxQ02Pushed PR branch to GitLabdevor PR is labeledno sandboxQ03Pushed PR branch to GitLabanvildevor PR is labeledno sandboxQ04Build passes insandboxdeployment or PR is labeledno sandboxQ05Build passes inanvilboxdeployment or PR is labeledno sandboxQ06Reviewed build logs for anomalies insandboxdeployment or PR is labeledno sandboxQ07Reviewed build logs for anomalies inanvilboxdeployment or PR is labeledno sandboxQ08Applied upgrade instructions from UPGRADING.rst tosandboxor this PR is not labeledupgrade, or upgrade instructions do not apply tosandboxQ09Applied upgrade instructions from UPGRADING.rst toanvilboxor this PR is not labeledupgrade, or upgrade instructions do not apply toanvilboxQ10Insandbox, deleted the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothQ11Inanvilbox, deleted the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothQ12Insandbox, deindexed the sources sepcified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothQ13Inanvilbox, deindexed the sources sepcified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothQ14Insandbox, indexed the sources specified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothQ15Inanvilbox, indexed the sources specified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothQ16Insandbox, indexed the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothQ17Inanvilbox, indexed the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothQ18Started full reindex insandboxor this PR is not labeledreindex:devor it is labeled reindex:partialQ19Started full reindex inanvilboxor this PR is not labeledreindex:anvildevor it is labeled reindex:partialQ20Checked for failures insandboxor this PR is not labeledreindex:devQ21Checked for failures inanvilboxor this PR is not labeledreindex:anvildevQ22Started mirroring insandboxor this PR is not labeledmirror:devQ23Started mirroring inanvilboxor this PR is not labeledmirror:anvildevQ24Checked for failures insandboxor this PR is not labeledmirror:devQ25Checked for failures inanvilboxor this PR is not labeledmirror:anvildevOperator (merge the branch)
R01All status checks passed and the PR is mergeableR02The title of the merge commit starts with the title of this PRR03Added PR # reference to merge commit titleR04Collected commit title tags in merge commit title but only includedpif the PR is also labeledpartialR05Pushed merge commit to GitHubR06Status of PR is Merged lowerR07Status of blocked issues is Triage or no issues are blocked on the linked issuesOperator (main build)
S01Pushed merge commit to GitLabdevS02Pushed merge commit to GitLabanvildevS03Build passes on GitLabdevS04Reviewed build logs for anomalies on GitLabdevS05Build passes on GitLabanvildevS06Reviewed build logs for anomalies on GitLabanvildevS07Applied upgrade instructions from UPGRADING.rst todevor this PR is not labeledupgrade, or upgrade instructions do not apply todevS08Applied upgrade instructions from UPGRADING.rst toanvildevor this PR is not labeledupgrade, or upgrade instructions do not apply toanvildevS09Notified developers to apply upgrade instructions from UPGRADING.rst to their personal deployments or this PR is not labeledupgrade, or upgrade instructions do not apply to personal deploymentsS10Ran_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:sharedS11Ran_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:sharedS12Deleted PR branch from GitHubS13PR is assigned to only the operatorS14Deleted PR branch from GitLabdevS15Deleted PR branch from GitLabanvildevS16Status of linked issues is Lower, or Triage, if PR is partialOperator (reindex)
T01Indev, deleted the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothT02Inanvildev, deleted the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothT03Indev, deindexed the sources sepcified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothT04Inanvildev, deindexed the sources sepcified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothT05Indev, indexed the sources specified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothT06Inanvildev, indexed the sources specified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothT07Indev, indexed the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:devlabel, or bothT08Inanvildev, indexed the catalogs specified in the notes or this PR is missing either thereindex:partialor thereindex:anvildevlabel, or bothT09Started full reindex indevor this PR is not labeledreindex:devor it is labeled reindex:partialT10Started full reindex inanvildevor this PR is not labeledreindex:anvildevor it is labeled reindex:partialT11Checked for, triaged and possibly requeued messages in both fail queues indevor this PR is not labeledreindex:devor it is labeled reindex:partialT12Checked for, triaged and possibly requeued messages in both fail queues inanvildevor this PR is not labeledreindex:anvildevor it is labeled reindex:partialT13Emptied fail queues indevor this PR is not labeledreindex:devor it is labeled reindex:partialT14Emptied fail queues inanvildevor this PR is not labeledreindex:anvildevor it is labeled reindex:partialT15Restarted the Data Browser pipeline for the ucsc/hca/dev branch on GitLab indev, and it succeeded or this PR is not labeledreindex:devT16Restarted the Data Browser pipeline for the ucsc/lungmap/dev branch on GitLab indev, and it succeeded or this PR is not labeledreindex:devT17Restarteddeploy_browserjob in the GitLab pipeline for this PR indev, and it succeeded or this PR is not labeledreindex:devT18Restarted the Data Browser pipeline for the ucsc/anvil/anvildev branch on GitLab inanvildev, and it succeeded or this PR is not labeledreindex:anvildevT19Restarteddeploy_browserjob in the GitLab pipeline for this PR inanvildev, and it succeeded or this PR is not labeledreindex:anvildevOperator (mirroring)
U01Started mirroring indevor this PR is not labelledmirror:devU02Started mirroring inanvildevor this PR is not labelledmirror:anvildevU03Checked for, triaged and possibly requeued messages in mirror fail queue indevor this PR is not labelledmirror:devU04Checked for, triaged and possibly requeued messages in mirror fail queue inanvildevor this PR is not labelledmirror:anvildevU05Emptied mirror fail queue indevor this PR is not labelledmirror:devU06Emptied mirror fail queue inanvildevor this PR is not labelledmirror:anvildevOperator
V01Propagated theupgrade,API,deploy:shared,deploy:gitlab,deploy:runner,reindex:partial,reindex:anvilprod,reindex:prod,mirror:partial,mirror:anvilprodandmirror:prodlabels to any open promotion PRs or this PR carries none of these labels, or is not included in an open promotion PRV02Propagated any specific instructions related to those labels, from the description of this PR to that of any open promotion PRs or this PR carries none of those labels, or is not included in an open promotion PRV03PR is assigned to no oneShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem