Promotion 2026-10-20 anvilprod (#8420) by achave11-ucsc · Pull Request #8429 · DataBiosphere/azul

achave11-ucsc · 2026-10-22T22:03:36Z

Linked issue: #8420

Notes

Perform a manual deindex of the following snapshots in anvil15:
- GTEx_V7_hg19_20221128_ANV5_202402291034
- GTEx_V8_hg38_20240116_ANV5_202409251632
- GTEx_V9_hg38_20221128_ANV5_202402070108
- GTEx_public_data_20240117_ANV5_202510101251
- GTEx_v10_hg38_20241105_ANV5_202511210107
- GTEx_v11_hg38_20260126_ANV5_202601261749
Perform a source specific reindex of the following snapshots in anvil15:
- GTEx_V7_hg19_20260821_ANV6_202608241432
- GTEx_V8_hg38_20240116_ANV6_202608211533
- GTEx_V9_hg38_20260821_ANV6_202608251459
- GTEx_public_data_20260821_ANV6_202608211401
- GTEx_v10_hg38_20260821_ANV6_202608241349
- GTEx_v11_hg38_20260126_ANV6_202608211606
Mirror anvil15 in anvilprod

Checklist

Author

Author (reindex)

C01 This PR is labeled reindex:anvilprod _{or the changes introduced by it will not require reindexing of anvilprod}
C02 This PR is labeled reindex:partial and its description documents the specific reindexing procedure for anvilprod _{or requires a full reindex or is not labeledreindex:anvilprod}

Author (mirror)

D01 This PR is labeled mirror:anvilprod _{or the changes introduced by it will not require mirroring of anvilprod}
D02 This PR is labeled mirror:partial and its description documents the specific mirroring procedure for anvilprod _{or requires a full mirroring or is not labeledmirror:anvilprod}

Author (upgrading deployments)

F01 This PR is labeled upgrade _{or does not require upgrading deployments}
F02 This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
F03 This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
F04 This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

H01 PR branch is up to date (if not, merge anvilprod into PR branch to integrate upstream changes)
H02 PR is not a draft
H03 PR is awaiting requested review from system administrator
H04 Status of PR is Review requested
H05 PR is assigned to only the system administrator and the author

System administrator (after approval)

K01 Actually approved the PR
K02 Decided if PR can be labeled no sandbox
K03 N reviews label is accurate
K04 Status of PR is Approved
K05 PR is assigned to only the operator and the author

Operator

L01 Pushed PR branch to GitHub

Operator (deploy `.shared` and `.gitlab` components)

M01 Ran _select anvilprod.shared && CI_COMMIT_REF_NAME=anvilprod make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
M02 Ran _select anvilprod.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
M03 Ran _select anvilprod.gitlab && CI_COMMIT_REF_NAME=anvilprod make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) _{or this PR is not labeled deploy:gitlab}
M04 Checked the items in the next section _{or this PR is labeled deploy:gitlab}
M05 PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of `.gitlab` component)

N01 Background migrations for anvilprod.gitlab are complete _{or this PR is not labeled deploy:gitlab}
N02 PR is assigned to only the operator and the author

Operator (deploy runner image)

P01 Ran _select anvilprod.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (sandbox build)

Operator (merge the branch)

R01 All status checks passed and the PR is mergeable
R02 The title of the merge commit starts with the title of this PR
R03 Added PR # reference to merge commit title
R04 Collected commit title tags in merge commit title _{but excluded any p tags}
R05 Pushed merge commit to GitHub
R06 Status of PR is Merged stable

Operator (main build)

¹ Promoted issues and PRs are referenced in the titles of the commits
that the promotion branch introduces to the stable branch. Prior to the
promotion, the status of promoted issues (PRs) is Lower (Merged lower).
Promoted PRs in status Done do not need to be moved.

Operator (reindex)

T01 In anvilprod, deleted the catalogs specified in the notes _{or this PR is missing either the reindex:partial or the reindex:anvilprod label, or both}
T02 In anvilprod, deindexed the sources sepcified in the notes _{or this PR is missing either the reindex:partial or the reindex:anvilprod label, or both}
T03 In anvilprod, indexed the sources specified in the notes _{or this PR is missing either the reindex:partial or the reindex:anvilprod label, or both}
T04 In anvilprod, indexed the catalogs specified in the notes _{or this PR is missing either the reindex:partial or the reindex:anvilprod label, or both}
T05 Started full reindex in anvilprod _{or this PR is not labeled reindex:anvilprod or it is labeled reindex:partial}
T06 Checked for, triaged and possibly requeued messages in both fail queues in anvilprod _{or this PR is not labeled reindex:anvilprod or it is labeled reindex:partial}
T07 Emptied fail queues in anvilprod _{or this PR is not labeled reindex:anvilprod or it is labeled reindex:partial}
T08 Restarted the Data Browser pipeline for the ucsc/anvil/anvilprod branch on GitLab in anvilprod, and it succeeded _{or this PR is not labeled reindex:anvilprod}
T09 Restarted deploy_browser job in the GitLab pipeline for this PR in anvilprod, and it succeeded _{or this PR is not labeled reindex:anvilprod}

Operator (mirroring)

U01 Started mirroring in anvilprod _{or this PR is not labelled mirror:anvilprod}
U02 Checked for, triaged and possibly requeued messages in mirror fail queue in anvilprod _{or this PR is not labelled mirror:anvilprod}
U03 Emptied mirror fail queue in anvilprod _{or this PR is not labelled mirror:anvilprod}

Operator

V01 PR is assigned to only the system administrator

System administrator

W01 Removed unused image tags from pycharm image on DockerHub _{or this promotion does not alter references to that image}
W02 Removed unused image tags from bigquery_emulator image on DockerHub _{or this promotion does not alter references to that image}
W03 PR is assigned to no one

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The `deploy_site_browser` provisioner extracts the Data Browser tarball with `tar -xvjf`, which execs `bzip2`. The binary used to be present as a transitive dependency of `build-essential`, and left the image when that package started being purged after use. Nothing referenced it by name, so the loss was silent. It surfaced only now because the provisioner execs `bzip2` solely when there is a new tarball to extract, which is what a reindex of `anvildev` produced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The indexer SA lost access to 22 of the 25 sources in the lm10 catalog, which fails the deploy step during the outsourcing of sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

, PR #8411)

The `stitched` attribute of a canned HCA bundle is a set of the UUIDs of the entities that were stitched into that bundle. The canned TDR HCA bundle 4426adc5 also listed, for each stitched file, the UUID of that file's manifest entry. The plugin only ever adds an entity's own UUID, so the canner cannot have produced those six surplus entries. They were inert, because `Bundle.not_stitched` matches the set against each entity's `document_id`, which a manifest entry UUID never equals. The surplus went unnoticed because `_test_fetch_bundle` compared every other attribute of a canned bundle against that of the emulated one, but not `stitched`. It now does. The canned DSS bundles carry the same surplus, tracked in #8423. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The comment stated that the manifest and the metadata of a bundle are both sorted by entity UUID. The assertions below it compare dictionaries, which are equal regardless of the order of their entries, so nothing verified that claim. The claim was inaccurate, too. The entries are grouped by entity type and only sorted by UUID within such a group. The order of the groups is that of a set's iteration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A plugin groups those entries by entity type, in the iteration order of a set, which varies with Python's hash seed. Canning one bundle twice could therefore yield two files that differ only in the order of their entries. The canner now sorts both by entity reference. The entries of the canned TDR HCA bundles are reordered accordingly. They are otherwise unchanged, and so are the other attributes of those bundles. Only the canned file is normalized. The order of the entries of a bundle in memory still varies between processes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The canned bundles of `test_subgraph_stitching` formed a chain, so the plugin retrieved them one at a time, in the order the chain dictated. The order of the merged links only varies when one batch holds more than one bundle, a shape that fixture could not produce. The root bundle now consumes a second file, produced by a new bundle whose own input is a cell suspension rather than a file. That bundle is therefore self-contained, and the traversal retrieves it together with the first upstream bundle, in one batch. A batch is a set, whose iteration order varies between processes, so this test now fails in roughly half of all runs, until the commit that merges the links in a deterministic order. The new bundle, its process and the file it produces are synthetic. All other entities they reference already exist in the canned tables. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The stitched bundles of one batch are the elements of a set, so the order in which the plugin emits them, and with it the order of the merged links, varies between processes. The preceding commit made the fixture produce such a batch, but a single run of the test still covers only whichever order that process happens to produce. The new test merges the links of the stitched bundles in every permutation and expects the result of the canned bundle each time. It fails until the next commit, which merges them in a deterministic order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The TDR HCA plugin merged the links in the order in which it retrieved the stitched bundles. That is the iteration order of a set and differs between indexer invocations, so every partition of a stitched bundle could produce a links replica with a different content hash. In prod, one bundle has 101 replicas that differ only in the order of their links. The links of the root bundle now come first, followed by those of the other bundles in the order of their UUID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A test used to drop the tables it had created, so that the next test of the class could create them again in the same emulator. Dropping them grew more expensive the longer that emulator had been in use, from 4.7s in the first test of a class to 11.6s in the fifth, and dominated the run time of both affected test classes. A test now gets an emulator of its own and discards it afterwards. Launching one costs 0.3s and makes dropping the tables unnecessary. Running test_tdr and test_anvil together takes 59s instead of 142s, for the same 29 tests. Replacing the nineteen table deletions with a single deletion of the dataset was tried first. That deletion degrades just like the ones it replaces, so it saved only 12%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… differ only in link order (#8393, #8259, #8294, PR #8418)

coveralls · 2026-10-22T22:14:47Z

coverage: 85.074% (+0.01%) from 85.064% — promotions/2026-10-20-anvilprod into anvilprod

codecov · 2026-10-22T22:20:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.00%. Comparing base (40506e5) to head (5d9c1a5).

Additional details and impacted files

@@            Coverage Diff             @@
##           anvilprod    #8429   +/-   ##
==========================================
  Coverage      84.99%   85.00%           
==========================================
  Files            170      170           
  Lines          26068    26085   +17     
==========================================
+ Hits           22156    22173   +17     
  Misses          3912     3912

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hannes-ucsc and others added 23 commits September 24, 2026 15:15

[u r a A] Promotion 2026-09-22 prod (#8317, PR #8328)

023da22

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Merge branch 'prod' into promotions/2026-09-29-prod

c50bf5f

[u] Promotion 2026-09-29 prod (#8342, PR #8350)

45025ec

Merge branch 'prod' into promotions/2026-10-06-prod

008e4e6

[R] Promotion 2026-10-06 prod (#8373, PR #8379)

69da073

Merge branch 'prod' into promotions/2026-10-13-prod

37a0f48

[a] Promotion 2026-10-13 prod (#8390, PR #8406)

16f540d

[h] Workaround: Inaccessible LungMAP snapshots in prod (#8410)

7737c07

The indexer SA lost access to 22 of the 25 sources in the lm10 catalog, which fails the deploy step during the outsourcing of sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add FIXME (#8412)

5270454

[h] Workaround: Twenty-two inaccessible LungMAP snapshots in prod (#8410

74d2195

, PR #8411)

[r] Index GTEX snapshots held back from anvil15 (#8392)

e59b94b

[r] Index GTEX snapshots held back from anvil15 (#8392, PR #8408)

472daab

Backport: 74d2195 (#8410, PR #8411, PR #8413)

ed32af2

[u] Four inaccessible LungMAP snapshots in dev (#8421, #8412)

5e6a3f1

[r] Fix: Stitched HCA bundles are indexed as many links replicas that…

8842489

… differ only in link order (#8393, #8259, #8294, PR #8418)

achave11-ucsc requested a review from hannes-ucsc as a code owner October 22, 2026 22:03

achave11-ucsc linked an issue Oct 22, 2026 that may be closed by this pull request

Promotion 2026-10-20 #8420

Open

8 tasks done

Merge branch 'anvilprod' into promotions/2026-10-20-anvilprod

5d9c1a5

achave11-ucsc self-assigned this Oct 22, 2026

achave11-ucsc assigned hannes-ucsc Oct 22, 2026

hannes-ucsc approved these changes Oct 23, 2026

View reviewed changes

hannes-ucsc added the no sandbox [process] PR will not be tested in the sandbox label Oct 23, 2026

hannes-ucsc removed their assignment Oct 23, 2026

hannes-ucsc added 0 reviews [process] Lead didn't request any changes and removed no sandbox [process] PR will not be tested in the sandbox labels Oct 23, 2026

achave11-ucsc added the sandbox [process] Resolution is being verified in sandbox deployment label Oct 23, 2026

achave11-ucsc merged commit 8c2f4f3 into anvilprod Oct 23, 2026
9 checks passed

achave11-ucsc deleted the promotions/2026-10-20-anvilprod branch October 24, 2026 02:26

Navigation Menu

Repository navigation

Promotion 2026-10-20 anvilprod (#8420) - #8429

Promotion 2026-10-20 anvilprod (#8420)#8429
achave11-ucsc merged 24 commits into
anvilprodfrom
promotions/2026-10-20-anvilprod

achave11-ucsc commented Oct 22, 2026 •

edited

Loading

Uh oh!

coveralls commented Oct 22, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

achave11-ucsc commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Checklist

Author

Author (reindex)

Author (mirror)

Author (upgrading deployments)

Author (before every review)

System administrator (after approval)

Operator

Operator (deploy .shared and .gitlab components)

System administrator (post-deploy of .gitlab component)

Operator (deploy runner image)

Operator (sandbox build)

Operator (merge the branch)

Operator (main build)

Operator (reindex)

Operator (mirroring)

Operator

System administrator

Shorthand for review comments

Uh oh!

coveralls commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 22, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

achave11-ucsc commented Oct 22, 2026 •

edited

Loading

Operator (deploy `.shared` and `.gitlab` components)

System administrator (post-deploy of `.gitlab` component)

coveralls commented Oct 22, 2026 •

edited

Loading