Promotion 2026-10-20 prod (#8420) by achave11-ucsc · Pull Request #8428 · DataBiosphere/azul

achave11-ucsc · 2026-10-22T21:54:27Z

Linked issue: #8420

Checklist

Author

Author (reindex)

C01 This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
C02 This PR is labeled reindex:partial and its description documents the specific reindexing procedure for prod _{or requires a full reindex or is not labeledreindex:prod}

Author (mirror)

D01 This PR is labeled mirror:prod _{or the changes introduced by it will not require mirroring of prod}
D02 This PR is labeled mirror:partial and its description documents the specific mirroring procedure for prod _{or requires a full mirroring or is not labeledmirror:prod}

Author (upgrading deployments)

F01 This PR is labeled upgrade _{or does not require upgrading deployments}
F02 This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
F03 This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
F04 This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

H01 PR branch is up to date (if not, merge prod into PR branch to integrate upstream changes)
H02 PR is not a draft
H03 PR is awaiting requested review from system administrator
H04 Status of PR is Review requested
H05 PR is assigned to only the system administrator and the author

System administrator (after approval)

K01 Actually approved the PR
K02 Labeled PR as no sandbox
K03 N reviews label is accurate
K04 Status of PR is Approved
K05 PR is assigned to only the operator and the author

Operator

L01 Pushed PR branch to GitHub

Operator (deploy `.shared` and `.gitlab` components)

M01 Ran _select prod.shared && CI_COMMIT_REF_NAME=prod make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
M02 Ran _select prod.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
M03 Ran _select prod.gitlab && CI_COMMIT_REF_NAME=prod make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) _{or this PR is not labeled deploy:gitlab}
M04 Checked the items in the next section _{or this PR is labeled deploy:gitlab}
M05 PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of `.gitlab` component)

N01 Background migrations for prod.gitlab are complete _{or this PR is not labeled deploy:gitlab}
N02 PR is assigned to only the operator and the author

Operator (deploy runner image)

P01 Ran _select prod.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (merge the branch)

R01 All status checks passed and the PR is mergeable
R02 The title of the merge commit starts with the title of this PR
R03 Added PR # reference to merge commit title
R04 Collected commit title tags in merge commit title _{but excluded any p tags}
R05 Pushed merge commit to GitHub
R06 Status of PR is Merged stable

Operator (main build)

¹ Promoted issues and PRs are referenced in the titles of the commits
that the promotion branch introduces to the stable branch. Prior to the
promotion, the status of promoted issues (PRs) is Lower (Merged lower).
Promoted PRs in status Done do not need to be moved.

Operator (reindex)

Operator (mirroring)

U01 Started mirroring in prod _{or this PR is not labelled mirror:prod}
U02 Checked for, triaged and possibly requeued messages in mirror fail queue in prod _{or this PR is not labelled mirror:prod}
U03 Emptied mirror fail queue in prod _{or this PR is not labelled mirror:prod}

Operator

V01 PR is assigned to only the system administrator

System administrator

W01 Removed unused image tags from pycharm image on DockerHub _{or this promotion does not alter references to that image}
W02 Removed unused image tags from bigquery_emulator image on DockerHub _{or this promotion does not alter references to that image}
W03 PR is assigned to no one

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

The `stitched` attribute of a canned HCA bundle is a set of the UUIDs of the entities that were stitched into that bundle. The canned TDR HCA bundle 4426adc5 also listed, for each stitched file, the UUID of that file's manifest entry. The plugin only ever adds an entity's own UUID, so the canner cannot have produced those six surplus entries. They were inert, because `Bundle.not_stitched` matches the set against each entity's `document_id`, which a manifest entry UUID never equals. The surplus went unnoticed because `_test_fetch_bundle` compared every other attribute of a canned bundle against that of the emulated one, but not `stitched`. It now does. The canned DSS bundles carry the same surplus, tracked in #8423. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The comment stated that the manifest and the metadata of a bundle are both sorted by entity UUID. The assertions below it compare dictionaries, which are equal regardless of the order of their entries, so nothing verified that claim. The claim was inaccurate, too. The entries are grouped by entity type and only sorted by UUID within such a group. The order of the groups is that of a set's iteration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A plugin groups those entries by entity type, in the iteration order of a set, which varies with Python's hash seed. Canning one bundle twice could therefore yield two files that differ only in the order of their entries. The canner now sorts both by entity reference. The entries of the canned TDR HCA bundles are reordered accordingly. They are otherwise unchanged, and so are the other attributes of those bundles. Only the canned file is normalized. The order of the entries of a bundle in memory still varies between processes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The canned bundles of `test_subgraph_stitching` formed a chain, so the plugin retrieved them one at a time, in the order the chain dictated. The order of the merged links only varies when one batch holds more than one bundle, a shape that fixture could not produce. The root bundle now consumes a second file, produced by a new bundle whose own input is a cell suspension rather than a file. That bundle is therefore self-contained, and the traversal retrieves it together with the first upstream bundle, in one batch. A batch is a set, whose iteration order varies between processes, so this test now fails in roughly half of all runs, until the commit that merges the links in a deterministic order. The new bundle, its process and the file it produces are synthetic. All other entities they reference already exist in the canned tables. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The stitched bundles of one batch are the elements of a set, so the order in which the plugin emits them, and with it the order of the merged links, varies between processes. The preceding commit made the fixture produce such a batch, but a single run of the test still covers only whichever order that process happens to produce. The new test merges the links of the stitched bundles in every permutation and expects the result of the canned bundle each time. It fails until the next commit, which merges them in a deterministic order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The TDR HCA plugin merged the links in the order in which it retrieved the stitched bundles. That is the iteration order of a set and differs between indexer invocations, so every partition of a stitched bundle could produce a links replica with a different content hash. In prod, one bundle has 101 replicas that differ only in the order of their links. The links of the root bundle now come first, followed by those of the other bundles in the order of their UUID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A test used to drop the tables it had created, so that the next test of the class could create them again in the same emulator. Dropping them grew more expensive the longer that emulator had been in use, from 4.7s in the first test of a class to 11.6s in the fifth, and dominated the run time of both affected test classes. A test now gets an emulator of its own and discards it afterwards. Launching one costs 0.3s and makes dropping the tables unnecessary. Running test_tdr and test_anvil together takes 59s instead of 142s, for the same 29 tests. Replacing the nineteen table deletions with a single deletion of the dataset was tried first. That deletion degrades just like the ones it replaces, so it saved only 12%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… differ only in link order (#8393, #8259, #8294, PR #8418)

codecov · 2026-10-22T22:03:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.00%. Comparing base (74d2195) to head (8842489).

Additional details and impacted files

@@           Coverage Diff           @@
##             prod    #8428   +/-   ##
=======================================
  Coverage   84.99%   85.00%           
=======================================
  Files         170      170           
  Lines       26068    26085   +17     
=======================================
+ Hits        22157    22174   +17     
  Misses       3911     3911

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

coveralls · 2026-10-22T22:03:44Z

coverage: 85.074% (+0.01%) from 85.064% — promotions/2026-10-20-prod into prod

achave11-ucsc and others added 12 commits October 20, 2026 06:27

[r] Index GTEX snapshots held back from anvil15 (#8392)

e59b94b

[r] Index GTEX snapshots held back from anvil15 (#8392, PR #8408)

472daab

Backport: 74d2195 (#8410, PR #8411, PR #8413)

ed32af2

[u] Four inaccessible LungMAP snapshots in dev (#8421, #8412)

5e6a3f1

[r] Fix: Stitched HCA bundles are indexed as many links replicas that…

8842489

… differ only in link order (#8393, #8259, #8294, PR #8418)

achave11-ucsc requested a review from hannes-ucsc as a code owner October 22, 2026 21:54

achave11-ucsc linked an issue Oct 22, 2026 that may be closed by this pull request

Promotion 2026-10-20 #8420

Open

8 tasks done

achave11-ucsc added the reindex:prod [process] PR requires reindexing prod label Oct 22, 2026

achave11-ucsc assigned hannes-ucsc Oct 22, 2026

achave11-ucsc self-assigned this Oct 22, 2026

hannes-ucsc approved these changes Oct 23, 2026

View reviewed changes

hannes-ucsc added no sandbox [process] PR will not be tested in the sandbox 0 reviews [process] Lead didn't request any changes labels Oct 23, 2026

hannes-ucsc removed their assignment Oct 23, 2026

achave11-ucsc merged commit 8cd5acd into prod Oct 23, 2026
21 checks passed

Navigation Menu

Repository navigation

Promotion 2026-10-20 prod (#8420) - #8428

Promotion 2026-10-20 prod (#8420)#8428
achave11-ucsc merged 12 commits into
prodfrom
promotions/2026-10-20-prod

achave11-ucsc commented Oct 22, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 22, 2026

Uh oh!

coveralls commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

achave11-ucsc commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Author

Author (reindex)

Author (mirror)

Author (upgrading deployments)

Author (before every review)

System administrator (after approval)

Operator

Operator (deploy .shared and .gitlab components)

System administrator (post-deploy of .gitlab component)

Operator (deploy runner image)

Operator (merge the branch)

Operator (main build)

Operator (reindex)

Operator (mirroring)

Operator

System administrator

Shorthand for review comments

Uh oh!

codecov Bot commented Oct 22, 2026

Codecov Report

Uh oh!

coveralls commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

achave11-ucsc commented Oct 22, 2026 •

edited

Loading

Operator (deploy `.shared` and `.gitlab` components)

System administrator (post-deploy of `.gitlab` component)