Skip to content

Fix: Sometimes manage_queues does not dump all available messages (#3663) - #8414

Merged
hannes-ucsc merged 95 commits into
developfrom
issues/hannes-ucsc/3663-manage-queues
Oct 22, 2026
Merged

hannes-ucsc merged 95 commits into
developfrom
issues/hannes-ucsc/3663-manage-queues

Conversation

@hannes-ucsc

@hannes-ucsc hannes-ucsc commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Linked issues: #3663

Checklist

Author

  • A01 PR is assigned to the author
  • A02 Status of PR is In progress
  • A03 PR is a draft
  • A04 Target branch is develop
  • A05 Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
  • A06 PR is linked to all issues it (partially) resolves
  • A07 Status of linked issues is In progress
  • A08 PR description links to linked issues
  • A09 PR title matches1 that of a linked issue or comment in PR explains why they're different
  • A10 PR title references all linked issues
  • A11 For each linked issue, there is at least one commit whose title references that issue

1 when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

  • B01 Added p tag to titles of partial commits
  • B02 This PR is labeled partial or completely resolves all linked issues
  • B03 This PR partially resolves each of the linked issues or does not have the partial label

Author (reindex)

  • C01 Added r tag to commit title or the changes introduced by this PR will not require reindexing of any deployment
  • C02 This PR is labeled reindex:dev or the changes introduced by it will not require reindexing of dev
  • C03 This PR is labeled reindex:anvildev or the changes introduced by it will not require reindexing of anvildev
  • C04 This PR is labeled reindex:anvilprod or the changes introduced by it will not require reindexing of anvilprod
  • C05 This PR is labeled reindex:prod or the changes introduced by it will not require reindexing of prod
  • C06 This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod

Author (mirror)

  • D01 This PR is labeled mirror:dev or the changes introduced by it will not require mirroring of dev
  • D02 This PR is labeled mirror:anvildev or the changes introduced by it will not require mirroring of anvildev
  • D03 This PR is labeled mirror:anvilprod or the changes introduced by it will not require mirroring of anvilprod
  • D04 This PR is labeled mirror:prod or the changes introduced by it will not require mirroring of prod
  • D05 This PR is labeled mirror:partial and its description documents the specific mirroring procedure for dev, anvildev, anvilprod and prod or requires a full mirroring or carries none of the labels mirror:dev, mirror:anvildev, mirror:anvilprod and mirror:prod

Author (API changes)

  • E01 This PR and its linked issues are labeled API or this PR does not modify a REST API
  • E02 Added a (A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST API
  • E03 Updated REST API version number in app.py or this PR does not modify a REST API

Author (upgrading deployments)

  • F01 Ran make docker_images.json and committed the resulting changes or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable
  • F02 Documented upgrading of deployments in UPGRADING.rst or this PR does not require upgrading deployments
  • F03 Added u tag to commit title or this PR does not require upgrading deployments
  • F04 This PR is labeled upgrade or does not require upgrading deployments
  • F05 This PR is labeled deploy:shared or does not modify docker_images.json, and does not require deploying the shared component for any other reason
  • F06 This PR is labeled deploy:gitlab or does not require deploying the gitlab component
  • F07 This PR is labeled deploy:runner or does not require deploying the runner image

Author (hotfixes)

  • G01 Added F tag to main commit title or this PR does not include permanent fix for a temporary hotfix
  • G02 Reverted the temporary hotfixes for any linked issues or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues linked to this PR

Author (before every review)

  • H01 Rebased PR branch on develop, squashed fixups from prior reviews
  • H02 Ran make requirements_update or this PR does not modify pyproject.toml
  • H03 Added R tag to commit title or this PR does not modify uv.lock
  • H04 This PR is labeled reqs or does not modify uv.lock
  • H05 make integration_test passes in personal deployment or this PR does not modify functionality that could affect the IT outcome
  • H06 PR is awaiting requested review from a peer
  • H07 Status of PR is Review requested
  • H08 PR is assigned to only the peer and the author

Peer reviewer (after approval)

Note that after requesting changes, the PR must be assigned to only the author.

  • J01 Actually approved the PR
  • J02 PR is not a draft
  • J03 PR is awaiting requested review from system administrator
  • J04 Status of PR is Review requested
  • J05 PR is assigned to only the system administrator and the author

System administrator (after approval)

  • K01 Actually approved the PR
  • K02 Labeled linked issues as demo or no demo
  • K03 Commented on linked issues about demo expectations or all linked issues are labeled no demo
  • K04 Decided if PR can be labeled no sandbox
  • K05 A comment to this PR details the completed security design review
  • K06 PR title is appropriate as title of merge commit
  • K07 N reviews label is accurate
  • K08 Status of PR is Approved
  • K09 PR is assigned to only the operator and the author

Operator

  • L01 Checked reindex:… labels and r commit title tag
  • L02 Checked mirror:… labels
  • L03 Checked that demo expectations are clear or all linked issues are labeled no demo
  • L04 Squashed PR branch and rebased onto develop
  • L05 Sanity-checked history
  • L06 Pushed PR branch to GitHub

Operator (deploy .shared and .gitlab components)

  • M01 Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • M02 Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) or this PR is not labeled deploy:gitlab
  • M03 Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused or this PR is not labeled deploy:shared
  • M04 Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) or this PR is not labeled deploy:gitlab
  • M05 Checked the items in the next section or this PR is labeled deploy:gitlab
  • M06 PR is assigned to only the system administrator and the author or this PR is not labeled deploy:gitlab

System administrator (post-deploy of .gitlab component)

  • N01 Background migrations for dev.gitlab are complete or this PR is not labeled deploy:gitlab
  • N02 Background migrations for anvildev.gitlab are complete or this PR is not labeled deploy:gitlab
  • N03 PR is assigned to only the operator and the author

Operator (deploy runner image)

  • P01 Ran _select dev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner
  • P02 Ran _select anvildev.gitlab && make -C terraform/gitlab/runner or this PR is not labeled deploy:runner

Operator (sandbox build)

  • Q01 Added sandbox label or PR is labeled no sandbox
  • Q02 Pushed PR branch to GitLab dev or PR is labeled no sandbox
  • Q03 Pushed PR branch to GitLab anvildev or PR is labeled no sandbox
  • Q04 Build passes in sandbox deployment or PR is labeled no sandbox
  • Q05 Build passes in anvilbox deployment or PR is labeled no sandbox
  • Q06 Reviewed build logs for anomalies in sandbox deployment or PR is labeled no sandbox
  • Q07 Reviewed build logs for anomalies in anvilbox deployment or PR is labeled no sandbox
  • Q08 Applied upgrade instructions from UPGRADING.rst to sandbox or this PR is not labeled upgrade, or upgrade instructions do not apply to sandbox
  • Q09 Applied upgrade instructions from UPGRADING.rst to anvilbox or this PR is not labeled upgrade, or upgrade instructions do not apply to anvilbox
  • Q10 In sandbox, deleted the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • Q11 In anvilbox, deleted the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • Q12 In sandbox, deindexed the sources sepcified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • Q13 In anvilbox, deindexed the sources sepcified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • Q14 In sandbox, indexed the sources specified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • Q15 In anvilbox, indexed the sources specified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • Q16 In sandbox, indexed the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • Q17 In anvilbox, indexed the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • Q18 Started full reindex in sandbox or this PR is not labeled reindex:dev or it is labeled reindex:partial
  • Q19 Started full reindex in anvilbox or this PR is not labeled reindex:anvildev or it is labeled reindex:partial
  • Q20 Checked for failures in sandbox or this PR is not labeled reindex:dev
  • Q21 Checked for failures in anvilbox or this PR is not labeled reindex:anvildev
  • Q22 Started mirroring in sandbox or this PR is not labeled mirror:dev
  • Q23 Started mirroring in anvilbox or this PR is not labeled mirror:anvildev
  • Q24 Checked for failures in sandbox or this PR is not labeled mirror:dev
  • Q25 Checked for failures in anvilbox or this PR is not labeled mirror:anvildev

Operator (merge the branch)

  • R01 All status checks passed and the PR is mergeable
  • R02 The title of the merge commit starts with the title of this PR
  • R03 Added PR # reference to merge commit title
  • R04 Collected commit title tags in merge commit title but only included p if the PR is also labeled partial
  • R05 Pushed merge commit to GitHub
  • R06 Status of PR is Merged lower
  • R07 Status of blocked issues is Triage or no issues are blocked on the linked issues

Operator (main build)

  • S01 Pushed merge commit to GitLab dev
  • S02 Pushed merge commit to GitLab anvildev
  • S03 Build passes on GitLab dev
  • S04 Reviewed build logs for anomalies on GitLab dev
  • S05 Build passes on GitLab anvildev
  • S06 Reviewed build logs for anomalies on GitLab anvildev
  • S07 Applied upgrade instructions from UPGRADING.rst to dev or this PR is not labeled upgrade, or upgrade instructions do not apply to dev
  • S08 Applied upgrade instructions from UPGRADING.rst to anvildev or this PR is not labeled upgrade, or upgrade instructions do not apply to anvildev
  • S09 Notified developers to apply upgrade instructions from UPGRADING.rst to their personal deployments or this PR is not labeled upgrade, or upgrade instructions do not apply to personal deployments
  • S10 Ran _select dev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • S11 Ran _select anvildev.shared && make -C terraform/shared apply or this PR is not labeled deploy:shared
  • S12 Deleted PR branch from GitHub
  • S13 PR is assigned to only the operator
  • S14 Deleted PR branch from GitLab dev
  • S15 Deleted PR branch from GitLab anvildev
  • S16 Status of linked issues is Lower, or Triage, if PR is partial

Operator (reindex)

  • T01 In dev, deleted the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • T02 In anvildev, deleted the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • T03 In dev, deindexed the sources sepcified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • T04 In anvildev, deindexed the sources sepcified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • T05 In dev, indexed the sources specified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • T06 In anvildev, indexed the sources specified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • T07 In dev, indexed the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:dev label, or both
  • T08 In anvildev, indexed the catalogs specified in the notes or this PR is missing either the reindex:partial or the reindex:anvildev label, or both
  • T09 Started full reindex in dev or this PR is not labeled reindex:dev or it is labeled reindex:partial
  • T10 Started full reindex in anvildev or this PR is not labeled reindex:anvildev or it is labeled reindex:partial
  • T11 Checked for, triaged and possibly requeued messages in both fail queues in dev or this PR is not labeled reindex:dev or it is labeled reindex:partial
  • T12 Checked for, triaged and possibly requeued messages in both fail queues in anvildev or this PR is not labeled reindex:anvildev or it is labeled reindex:partial
  • T13 Emptied fail queues in dev or this PR is not labeled reindex:dev or it is labeled reindex:partial
  • T14 Emptied fail queues in anvildev or this PR is not labeled reindex:anvildev or it is labeled reindex:partial
  • T15 Restarted the Data Browser pipeline for the ucsc/hca/dev branch on GitLab in dev, and it succeeded or this PR is not labeled reindex:dev
  • T16 Restarted the Data Browser pipeline for the ucsc/lungmap/dev branch on GitLab in dev, and it succeeded or this PR is not labeled reindex:dev
  • T17 Restarted deploy_browser job in the GitLab pipeline for this PR in dev, and it succeeded or this PR is not labeled reindex:dev
  • T18 Restarted the Data Browser pipeline for the ucsc/anvil/anvildev branch on GitLab in anvildev, and it succeeded or this PR is not labeled reindex:anvildev
  • T19 Restarted deploy_browser job in the GitLab pipeline for this PR in anvildev, and it succeeded or this PR is not labeled reindex:anvildev

Operator (mirroring)

  • U01 Started mirroring in dev or this PR is not labelled mirror:dev
  • U02 Started mirroring in anvildev or this PR is not labelled mirror:anvildev
  • U03 Checked for, triaged and possibly requeued messages in mirror fail queue in dev or this PR is not labelled mirror:dev
  • U04 Checked for, triaged and possibly requeued messages in mirror fail queue in anvildev or this PR is not labelled mirror:anvildev
  • U05 Emptied mirror fail queue in dev or this PR is not labelled mirror:dev
  • U06 Emptied mirror fail queue in anvildev or this PR is not labelled mirror:anvildev

Operator

  • V01 Propagated the upgrade, API, deploy:shared, deploy:gitlab, deploy:runner, reindex:partial, reindex:anvilprod, reindex:prod, mirror:partial, mirror:anvilprod and mirror:prod labels to any open promotion PRs or this PR carries none of these labels, or is not included in an open promotion PR
  • V02 Propagated any specific instructions related to those labels, from the description of this PR to that of any open promotion PRs or this PR carries none of those labels, or is not included in an open promotion PR
  • V03 PR is assigned to no one

Shorthand for review comments

  • L line is too long
  • W line wrapping is wrong
  • Q bad quotes
  • F other formatting problem

@hannes-ucsc hannes-ucsc self-assigned this Oct 17, 2026
@hannes-ucsc hannes-ucsc linked an issue Oct 17, 2026 that may be closed by this pull request
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.32258% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.42%. Comparing base (8842489) to head (9c31eed).

Files with missing lines Patch % Lines
src/azul/queues.py 79.21% 37 Missing ⚠️
src/azul/deployment.py 50.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8414      +/-   ##
===========================================
+ Coverage    85.00%   85.42%   +0.41%     
===========================================
  Files          170      171       +1     
  Lines        26085    26423     +338     
===========================================
+ Hits         22174    22571     +397     
+ Misses        3911     3852      -59     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coveralls

coveralls commented Oct 17, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 85.489% (+0.4%) from 85.078% — issues/hannes-ucsc/3663-manage-queues into develop

@hannes-ucsc
hannes-ucsc force-pushed the issues/hannes-ucsc/3663-manage-queues branch from 396f7c4 to 208aa31 Compare October 22, 2026 00:40
@hannes-ucsc

Copy link
Copy Markdown
Member Author

Security design review

  • Security design review completed; this PR does not …
    • … affect authentication; for example:
      • OAuth 2.0 with the application (API or Swagger UI)
      • Authentication of developers with Google Cloud APIs
      • Authentication of developers with AWS APIs
      • Authentication with a GitLab instance in the system
      • Password and 2FA authentication with GitHub
      • API access token authentication with GitHub
      • Authentication with Terra
    • … affect the permissions of internal users like access to
      • Cloud resources on AWS and GCP
      • GitLab repositories, projects and groups, administration
      • an EC2 instance via SSH
      • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
    • … affect the permissions of external users like access to
      • TDR snapshots
    • … affect permissions of service or bot accounts
      • Cloud resources on AWS and GCP
    • … affect audit logging in the system, like
      • adding, removing or changing a log message that represents an auditable event
      • changing the routing of log messages through the system
    • … affect monitoring of the system
    • … introduce a new software dependency like
      • Python packages on PYPI
      • Command-line utilities
      • Docker images
      • Terraform providers
    • … add an interface that exposes sensitive or confidential data at the security boundary
    • … affect the encryption of data at rest
    • … require persistence of sensitive or confidential data that might require encryption at rest
    • … require unencrypted transmission of data within the security boundary
    • … affect the network security layer; for example by
      • modifying, adding or removing firewall rules
      • modifying, adding or removing security groups
      • changing or adding a port a service, proxy or load balancer listens on
  • Documentation on any unchecked boxes is provided in comments below

hannes-ucsc and others added 24 commits October 21, 2026 21:56
The entity types in an AnVIL bundle are the names of the BigQuery tables the
entities are drawn from, and therefore carry an `anvil_` prefix. The set of
types to redact lists them without that prefix, so the membership test never
holds, and `--redaction-key` quietly writes the bundle out unredacted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bundle holds its orphans separately from its entities, and only the latter
were redacted, leaving an orphaned biosample, diagnosis or donor in the clear.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SQS redrive facility moves messages server-side, so unlike the dump
and feed commands it is not subject to the limitations of receiving from
a FIFO queue, where the messages of a group behind an in-flight head
remain unreachable.

The source queue must be configured as the dead-letter queue of another
queue, and both queues must be of the same type. The command waits for
the move to complete. Since SQS only reports the handle of a task while
that task is running, a finished task is identified by when it started.

Also make the optional arguments of the client method keyword-only,
matching how every call site already passes them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check reads the queue property of the dump envelope, which a later
commit replaces. Validation returns at the end of this branch, in terms
of the type of message rather than the identity of the queue, so that
feeding the contents of a fail queue into the corresponding work queue
no longer requires an override. The --force flag goes with the check,
and the procedure in OPERATOR.rst no longer refers to force-feeding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_dump_messages condensed the messages it was given, but the cleanup of
an interrupted feed passes it the already condensed messages it read
from the dump file. The resulting AttributeError was raised while
handling the exception that interrupted the feed, hiding the actual
cause of the interruption from the operator.

The messages are now condensed by _dump, the only caller that holds
uncondensed ones. Present since 2cf70b9.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cleanup of an interrupted feed rewrote the dump file from the
complete list of messages instead of the ones that had not been sent
yet, so feeding the file again sent those messages a second time. The
list of remaining messages was computed but only used to decide whether
any message had been sent at all. Present since 2cf70b9.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither command was covered by a test. One test round-trips the messages
of a queue through a dump file, the other interrupts a feed and covers
the two defects fixed in dcf0d854f and eea840207, by asserting that the
exception which interrupted the feed is the one that propagates, and
that the file is left holding exactly the messages that were not sent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Appending to a dump file is a prerequisite for deleting each batch of
messages as it is written, which is in turn what allows a FIFO queue to
be dumped completely. This commit only changes the format, leaving the
dump to collect every message in memory before writing it, and the feed
to read the whole file into memory.

The first line of a dump is the envelope, each subsequent line is one
message. Dumps written by earlier versions are no longer accepted, and
dump_all now names its files after the queue with a .jsonl suffix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A message held in flight blocks the rest of its group, so a FIFO queue
cannot be dumped completely unless each batch is deleted from the queue
before the next one is received. Each batch is now written to the dump
file and flushed to disk before it is deleted, so that an interrupted
dump leaves behind every message it removed from the queue.

This also keeps the number of in-flight messages at one batch, well
below the limit SQS imposes on a queue, and it bounds the time a message
is invisible by the duration of one batch instead of the whole dump.

Because the messages of a dump that does not delete them must be held in
flight until the dump ends, or receiving them again would make the dump
run forever, such a dump still captures only the messages at the head of
each group. It now says so before it starts.

The dump file is no longer written through write_file_atomically, whose
removal of the temporary file on failure would discard exactly those
messages that were already deleted from the queue.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A short poll samples a subset of the SQS hosts and can therefore return
nothing while messages remain, ending a dump prematurely. This affects
FIFO and standard queues alike. A long poll queries all hosts, and the
documented maximum wait is used because AWS notes that lower values
make empty responses more likely.

The wait is an attribute so that tests can disable it. Moto honours the
parameter by blocking, and every drain ends on an empty receive, which
would otherwise add twenty seconds to each one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wait after purging a queue added the number of available messages
twice and never counted the ones in flight, so it could return while
messages were still being processed, just before the readers of the
queue are enabled again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An empty receive ended a dump even when the queue still reported
messages. The dump now consults those counts, discounting the messages
it holds in flight itself, and keeps polling while they disagree with
the empty receive.

The counts are approximate, and the producers of a queue keep adding to
it unless they are disabled, so polling is bounded: after three
consecutive empty receives the dump gives up and says what the queue
still reports.

The names of the three counting attributes are now defined once, and
the wait after purging a queue uses them, too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The --no-json-body flag stored the body of a message verbatim instead of
embedding the JSON it contains. Nothing but the command line ever set
it, and every message azul queues carries a JSON body, so a body that
the flag was needed for could only come from outside azul.

The inverse operation loses the branch that re-serialized such a body,
which means a dump written with the flag can no longer be fed back.
Dumps written before this branch could not be, either, since the format
has changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The class is not slotted because its cached property caches by writing
to the instance dictionary, which a slotted class lacks. The sole field
keeps its leading underscore, so that attrs names the constructor
argument after it without one, as the callers already do.

Only the constant naming the attributes that count the messages in a
queue is annotated, so it is the only one attrs would have mistaken for
a field, and it is declared as a class variable to prevent that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two descriptions differed only in whether they spoke of one queue
and file or of several, which the wording now avoids. One of them also
said "the its file".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two subcommands defined the option identically, which left them free
to drift apart. They now inherit it from a parser that carries nothing
else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lambdas reading the work queues produce the messages in every queue,
so a dump taken while they are running is likely to miss messages. The
dump now consults the work queues, and if they hold messages, insists on
a decision: --safe disables those lambdas for its duration, --unsafe
accepts a dump that is probably incomplete. Combining --unsafe with
--delete is refused, because it would discard the messages the dump
missed.

A dump taken while the work queues are empty needs no decision, and only
warns that nothing kept messages from arriving while it ran.

The two flags write to one tri-state attribute, so they cannot both be
in effect. The mutually exclusive group they share only improves the
error message.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An interrupted feed rewrote the dump with the messages it had not
submitted, which both destroyed the dump it was fed from and left
nothing to say how far it had come. It now records that in a file beside
the dump, rewritten after every batch, and skips as many messages when
it is run again. The file names the queue it fed, so that a dump
interrupted on its way into one queue is not silently truncated on its
way into another.

The --delete flag now removes the dump once every message in it was
submitted, instead of removing them from it one batch at a time.

A crash between submitting a batch and recording it resubmits that batch
when the feed is resumed. On a FIFO queue, SQS discards the resulting
duplicates, because the deduplication IDs are carried over from the
dump. On a standard queue it does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dump now records the type of message it holds, and a feed refuses a
dump whose type is not the one the target queue is expected to hold.
There are three mutually incompatible types, each named after the class
whose instances are the bodies of those messages. Those classes are
defined in modules that import this one, hence the deferred import.

Unlike the check this replaces, which compared the queue a dump was
taken from against the one it is fed into, this one permits feeding the
contents of a fail queue into the work queue it belongs to, which is
what the operator manual prescribes. It has no override, because no
queue can do anything with a message of another type.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dump records the queue it was taken from, and a feed refuses a dump
whose deployment is not the current one, unless --foreign says to
proceed. Type compatibility does not catch this, because the messages of
one deployment are perfectly well-formed messages of another.

Moving messages between deployments is a legitimate thing to do, for
instance to reproduce a failure in a personal deployment, which is why
this check has an override while the one on the type of message does
not.

The queue is recorded by name rather than by URL, because the name is
what carries the deployment, and the URL adds an account and a region to
a file that operators read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dump used to replace the contents of the file it was given, silently
discarding an earlier dump. It now refuses that file unless --overwrite
says to replace it, or --append says to add to it. Appending compares
the header of the file against the one the dump would write, so that a
dump of one queue is not appended to a dump of another.

The two flags write to one tri-state attribute, as the flags deciding
what a dump does about the queue worker lambdas do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every condition the queue management refuses to work under is a demand
it makes of its input or of the state of the deployment, not a defect in
itself, which is what R is for.

Two of them were stated as branches of a chain that had no condition to
assert, and are now stated as the conditions under which a dump of a
busy deployment, and a dump to a file that exists, may proceed. The
chain deciding whether to append collapses as a result, the attribute
being a plain boolean once it is known not to be None.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The procedure spoke of notifications and had the operator repeat it for
tallies afterwards, as an aside. It now covers both at once, in numbered
steps, and prescribes feeding the tallies first, they being downstream
from the notifications. Between dumping and feeding, the operator is now
asked to triage the failures and to agree on a remedy, rather than to
feed the messages back as they are.

Messages are now fed into the work queue, with the retry queue reserved
for the case where most of them are known to need the memory and the
timeout of the retry lambda, instead of being fed into the retry queue
to save time.

The constraints the script enforces on either step are described as
well, along with the flags that relax those that may be relaxed.

The instructions for tracking the remaining failures in tickets are
removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hannes-ucsc and others added 25 commits October 21, 2026 21:56
Indexing a bare dict yields Any, which spreads to the body and the URL
of an existing PR. Typing the response as JSON makes that indexing yield
a JSON value, which json_str narrows to the string both of them are.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hannes-ucsc
hannes-ucsc force-pushed the issues/hannes-ucsc/3663-manage-queues branch from 208aa31 to 9c31eed Compare October 22, 2026 06:05
@hannes-ucsc hannes-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Oct 22, 2026
@hannes-ucsc
hannes-ucsc merged commit 2a07043 into develop Oct 22, 2026
10 checks passed
@hannes-ucsc
hannes-ucsc deleted the issues/hannes-ucsc/3663-manage-queues branch October 22, 2026 20:20
@hannes-ucsc hannes-ucsc removed their assignment Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sandbox [process] Resolution is being verified in sandbox deployment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sometimes manage_queues does not dump all available messages

3 participants