Skip to content

feat: build the tier 1 metadata dictionary as an xlsx download #3261 - #3279

Draft
MillenniumFalconMechanic wants to merge 3 commits into
mainfrom
millenniumfalconmechanic/3261-tier-1-csv-download
Draft

MillenniumFalconMechanic wants to merge 3 commits into
mainfrom
millenniumfalconmechanic/3261-tier-1-csv-download

Conversation

@MillenniumFalconMechanic

@MillenniumFalconMechanic MillenniumFalconMechanic commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #3261

Draft: nothing on the page links to the download yet. The button is tracked in DataBiosphere/findable-ui#1143 and waits on the design in clevercanary/cc-design#340. Because the file is now a static XLSX, the button only needs to link to /downloads/hca-tier-1-metadata.xlsx (see the comment on findable-ui#1143). This PR can merge once that ships in a findable-ui release and the dependency is bumped here.

What changed

At build time, a script reads the portal's tier-1.json and uses exceljs to write a one-row-per-field XLSX, laid out like the Adipose metadata sheet, which the site serves as a static download.

  • scripts/build-tier-1-metadata-xlsx.ts (new): writes public/downloads/hca-tier-1-metadata.xlsx.
    • Columns: #, Field name, Field programmatic name, Description, Data type, Categorical values, Examples, Required, Encoded by, BioNetworks, AnnData Location, Multivalued.
    • Required uses the chip's bionetwork-scoped label (e.g. "Required (Gut)") via the exported buildRequiredLabel.
    • BioNetworks shows "ALL" when a field applies to every bionetwork.
    • Descriptions and categorical values are converted from Markdown to plain, single-line text by parsing them with remark (remark-parse + remark-gfm, already dependencies). Links keep their text, followed by the URL in brackets unless it points within the page. The one table (in X; raw.X) is replaced by "(See the table at …)" with a link to that field in the live dictionary.
    • Formatting: a bold, frozen header row and fixed column widths. No colours and no wrapped cells.
  • scripts/build-tier-1-metadata-xlsx.sh (new): bundles the script with esbuild and pipes it into Node. Plain Node can't load findable-ui's extensionless imports.
  • scripts/build.sh and the dev script in package.json: run the generator, so every build and every npm run dev produces the file.
  • .gitignore: ignores the generated public/downloads/.
  • package.json: adds the dev dependencies exceljs (build-time only, nothing ships to the browser) and esbuild (previously only a dependency of esrun).
  • Removed: the earlier browser-side TSV download (tableDownload.ts and the Tier 1 onTableDownload table options).

Why

Users asked for a downloadable copy of the Tier 1 metadata dictionary to work with offline and share while preparing their data. A spreadsheet file, rather than TSV, lets us set column widths and a frozen header, and opens directly in Excel and Google Sheets.

Assumptions I made

  • Built at build time, served statically. The site is a static export, so the file is generated before next build and served from /downloads/ rather than created in the browser. Generating it from tier-1.json on every build keeps it from drifting.
  • Full dictionary only. The file always contains all 51 Tier 1 fields. It doesn't follow the page's search or filters, which is one of the open questions in cc-design#340.
  • Layout follows the Adipose Tier 2 sheet for column names and order. Its "Imported from bionetwork" and "Public / DB side" columns have no equivalent in our data and are left out. BioNetworks, AnnData Location and Multivalued are added at the end.
  • "Encoded by" holds the class title (Dataset, Donor, Sample…).
  • No colour. The Adipose sheet colours rows by topic, but our class is already in "Encoded by", so colours would only repeat it.
  • Single-line cells. Line breaks in descriptions become spaces, and long text is clipped at the column width. Clicking a cell shows the full text.
  • Table links point at the live portal (https://data.humancellatlas.org/metadata/tier-1#<field>), including in files from dev builds.
  • Rationale and Source aren't included. They aren't in the issue's column list.
  • Tier 1 only. The issue's "consider Tier 2 and Cell Annotation" item is out of scope for this PR.

How to verify

  1. Run npm ci && npm run dev. The console prints Wrote public/downloads/hca-tier-1-metadata.xlsx (51 fields) before Next starts.
  2. Open http://localhost:3000/downloads/hca-tier-1-metadata.xlsx. It downloads an XLSX. Open it in Excel, or upload it to Drive and open it with Google Sheets.
  3. The sheet "Tier 1 metadata" has a bold header row that stays in place when scrolling, plus 51 rows with the 12 columns above, in a fixed width with no wrapping and no colours.
  4. Required shows "Required (Adipose, Gut, Musculoskeletal)" for ambient_count_correction and doublet_detection, "Required (Gut)" for radial_tissue_term and "Required (Adipose, Gut)" for dissociation_protocol. BioNetworks shows "ALL" for the other 47 fields.
  5. Descriptions read as plain text, e.g. author_cell_type: "Encoding of author intuition of cellular annotation in the dataset. Use author_cell_type for original source datasets where cell_type_ontology_term_id is not available."
  6. npm run build-prod:data-portal produces out/downloads/hca-tier-1-metadata.xlsx.
  7. Once findable-ui#1143 is available: on /metadata/tier-1, the download control fetches this file.

Definition of done from #3261:

  • The Tier 1 dictionary page has a download action that produces a spreadsheet → step 7. Waits for findable-ui#1143; the file itself is in this PR (steps 1–6).
  • One row per field, with the listed columns, requirement level including bionetwork scope → steps 3–4
  • The file is built from the same tier-1.json source, so it can't drift from the page → steps 1 and 6 (regenerated on every dev start and build)
  • Consider giving Tier 2 and Cell Annotation dictionaries the same action → out of scope (Tier 1 only, as agreed)

Screenshots

  • data-portal-3261-sample-hca-tier-1-metadata.xlsx: the generated file, for reviewers to open (no on-screen change to capture yet).

data-portal-3261-sample-hca-tier-1-metadata.xlsx

🤖 Generated with Claude Code
🐦 Built with cc-claude-tools

- Custom onTableDownload for the Tier 1 table writes one TSV row per field
- Requirement column uses the chip's bionetwork-scoped label
- UTF-8 byte order mark so Excel reads non-ASCII characters
- Download button tracked in DataBiosphere/findable-ui#1143

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

End-to-end verification remains pending the upstream download button release and required dependency bump.

No open issues 🎉

What changed in this PR

Adds TSV export support for the full Tier 1 metadata dictionary. The user-facing download action remains pending upstream integration.

Changes:

  • Builds an 11-column TSV with a UTF-8 byte order mark.
  • Configures Tier 1 table download handling.
  • Reuses bionetwork-scoped requirement labels.
File Description
viewModelBuilders/​dataDictionaryMapper/​tableOptions.ts Configures the Tier 1 download handler.
viewModelBuilders/​dataDictionaryMapper/​tableDownload.ts Generates and downloads the dictionary TSV.
components/​DataDictionary/​components/​TableCell/​components/​FieldCell/​utils.ts Exports the shared requirement-label helper.

Review effort: Balanced

…3261

- Script writes public/downloads/hca-tier-1-metadata.xlsx from tier-1.json with exceljs
- Columns follow the Adipose metadata sheet layout, one row per field
- Runs in scripts/build.sh and npm run dev; the generated file is git-ignored
- Replaces the browser-side TSV download

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
@MillenniumFalconMechanic MillenniumFalconMechanic changed the title feat: add tier 1 metadata dictionary tsv download #3261 feat: build the tier 1 metadata dictionary as an xlsx download #3261 Oct 21, 2026
@MillenniumFalconMechanic
MillenniumFalconMechanic requested a balanced review from Copilot October 21, 2026 22:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Markdown conversion alters existing dictionary guidance, and the stated download-control integration prerequisite remains outstanding.

1 open issue

Review effort: Balanced

Comment thread scripts/build-tier-1-metadata-xlsx.ts Outdated
…3261

- Parse descriptions and allowed values with remark instead of regexes
- Literal asterisks such as EGAX* are kept, and long emphasis is removed
- Links keep their text, followed by the URL unless it points within the page
- Tables are replaced by a link to the field's row in the dictionary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The documented merge prerequisite—the findable-ui download control release and dependency bump—requires final human verification.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: offer the Tier 1 metadata dictionary as a downloadable spreadsheet

2 participants