Scope: Only DPS jobs that produce a catalog.json referencing STAC Items have their outputs cataloged. Jobs do not need to supply a Collection object: when configured with CREATE_COLLECTIONS_IF_MISSING, stac-loader creates a placeholder for an Item's Collection ID if that Collection does not already exist. It does not generate the Items themselves.
MAAP users should be able to find DPS outputs by what they contain, how they were produced, and which work they belong to, without knowing an S3 path or decoding a Collection ID. This proposal describes the organization and metadata that would support that experience. Feedback on the user workflows is welcome; these are proposed capabilities, not a description of everything available today.
Today: Collections grouped by job tag
DPS outputs currently receive an automatically assigned Collection ID:
{username}__{algorithm_name}__{algorithm_version}__{job_tag}
These collections are stored in the DPS User STAC: https://dps-stac.maap-project.org/collections.
Different tags create separate Collections, even when the outputs otherwise belong together. For missing Collections, the STAC ingestion process creates a placeholder with the ID as its description, a proprietary license. It does not derive those values or algorithm summaries from the Items, and this fallback leaves existing Collections unchanged. Users therefore have little descriptive context beyond the Collection ID. The collection spatial/temporal extent is updated based on the item extents in a separate process.
The current structure makes it possible to perform STAC searches of DPS outputs in downstream workflows and to visualize DPS outputs as mosaics using titiler (example)
Proposed organization
In STAC, an Item describes an individual output and links to its files; a Collection groups related Items; a Catalog provides a browsing view of Collections.
We propose two ways to group DPS outputs:
- Automatic Collections: remove
job_tag from the default ID, giving {username}__{algorithm_name}__{algorithm_version}. Keep the tag on each Item so users can still find outputs from a particular experiment or submission grouping. This is important for multiple reasons:
- job tag is not a required field
- there are no rules about how to structure job tags - sometimes it is used to identify groups of jobs, sometimes it is used to identify individual jobs
- Named Collections: let users select a user- or group-owned Collection when submitting jobs, subject to publishing permissions. For example,
boreal-biomass-experiment-202609 could bring together outputs from several jobs, contributors, or algorithm versions.
Collections would appear in user-specific and, eventually, group-specific catalogs. The same Collection could appear under both “Alice’s Collections” and “Boreal Biomass Team” without duplicating its Items or files. Appearing in a group catalog would not transfer ownership or grant permission to publish.
Metadata for Items and Collections
The following fields are proposed discovery metadata (in progress here: MAAP-Project/maap-dps-stac-extension#2). We should reuse existing STAC fields where they fit and reserve custom fields for DPS-specific information.
All of these fields are derived from the metdata files that are staged out with every DPS job (i.e. *.met.json, *.context.json0).
Items: describe each output
| Information |
Proposed STAC field |
Example or meaning |
| Algorithm name |
maap-dps:algorithm_name |
esa-biomass-gamma0-fetch; the registered DPS algorithm that produced the output. |
| Algorithm version |
processing:version |
1.0.0; the version of that same algorithm or processing chain. |
| Job submitter |
maap-dps:username |
alice; distinct from Collection ownership. |
| Job tag |
maap-dps:tag |
experiment-42, when supplied. |
| Processing time |
processing:datetime |
When the data was processed, if known. |
| Supporting provenance |
Assets and links |
The job metadata file, upstream inputs (derived_from), and an execution record (processing-execution) when available. |
Keep searchable metadata on every Item even when its Collection contains only one algorithm/version. This makes searches independent of Collection naming and grouping.
The Processing extension already defines processing:version, but lacks a corresponding primary software/chain name. We are considering an upstream request for processing:name. Until such a field exists, retain maap-dps:algorithm_name; do not use free-text processing:lineage as an identifier.
Collections: describe the group of outputs
| Information |
Proposed STAC field or representation |
Example or meaning |
| Identity and description |
id, title, description |
A stable ID plus a readable title and explanation of the Collection's purpose. |
| Geographic and temporal coverage |
extent.spatial, extent.temporal |
The coverage of the Collection's Items. |
| Algorithm names represented |
summaries["maap-dps:algorithm_name"] |
One name for a homogeneous Collection; multiple names when appropriate. |
| Algorithm versions represented |
summaries["processing:version"] |
Versions represented by the Items. |
| Attribution and reuse |
providers, license, relevant links |
Who produced the data and how it may be used. |
| User/group association |
Catalog membership; separate MAAP ownership records |
Where the Collection appears for browsing, distinct from who may manage or publish to it. |
Collection summaries describe the values found across Items. They should reflect the contents rather than values inferred from the Collection ID. Job tags would remain Item-level metadata, not part of this Collection metadata proposal.
What users could find
- Browse: start from a user's or group's catalog to find related Collections.
- Search Collections: find Collections representing a particular algorithm or version, with geographic and temporal coverage of interest.
- Search Items: find Alice's
experiment-42 outputs from algorithm version 1.0.0, within a region or time period, across Collections.
Collection-level metadata would support Collection Search; Item properties would support targeted Item Search through the STAC API Filter extension (CQL2). Users would not need to know Collection IDs in advance.
One caveat: a Collection containing algorithm A/version 1 and algorithm B/version 2 summarizes both names and both versions. Those summaries alone cannot establish that it contains A/version 2. Exact algorithm/version matching in mixed Collections needs Item-level filtering, or future Collection search behavior that checks matching Items.
Related work
Richer Collection descriptions and searchable summaries depend on future improvements to the boilerplate Collection objects created by the STAC ingestion process. Those improvements can follow the initial change to tag-free Collection IDs and Item metadata.
The interfaces for creating Collections, selecting them during job submission, and managing ownership are a separate effort described in the DPS STAC integration planning documents. User catalogs are the initial browsing focus; group catalogs and group-owned Collections are part of the broader vision. Catalog browsing also does not automatically provide search scoped to a catalog.
Feedback requested
- Does removing the job tag from automatic Collection IDs fit your workflow?
- What would you group into a named Collection, and would it be user- or group-owned?
- Would user/group catalogs help you browse, and what searches or metadata are missing from this proposal?
cc @sujen1412 @wildintellect
Scope: Only DPS jobs that produce a
catalog.jsonreferencing STAC Items have their outputs cataloged. Jobs do not need to supply a Collection object: when configured withCREATE_COLLECTIONS_IF_MISSING,stac-loadercreates a placeholder for an Item's Collection ID if that Collection does not already exist. It does not generate the Items themselves.MAAP users should be able to find DPS outputs by what they contain, how they were produced, and which work they belong to, without knowing an S3 path or decoding a Collection ID. This proposal describes the organization and metadata that would support that experience. Feedback on the user workflows is welcome; these are proposed capabilities, not a description of everything available today.
Today: Collections grouped by job tag
DPS outputs currently receive an automatically assigned Collection ID:
These collections are stored in the DPS User STAC: https://dps-stac.maap-project.org/collections.
Different tags create separate Collections, even when the outputs otherwise belong together. For missing Collections, the STAC ingestion process creates a placeholder with the ID as its description, a
proprietarylicense. It does not derive those values or algorithm summaries from the Items, and this fallback leaves existing Collections unchanged. Users therefore have little descriptive context beyond the Collection ID. The collection spatial/temporal extent is updated based on the item extents in a separate process.The current structure makes it possible to perform STAC searches of DPS outputs in downstream workflows and to visualize DPS outputs as mosaics using titiler (example)
Proposed organization
In STAC, an Item describes an individual output and links to its files; a Collection groups related Items; a Catalog provides a browsing view of Collections.
We propose two ways to group DPS outputs:
job_tagfrom the default ID, giving{username}__{algorithm_name}__{algorithm_version}. Keep the tag on each Item so users can still find outputs from a particular experiment or submission grouping. This is important for multiple reasons:boreal-biomass-experiment-202609could bring together outputs from several jobs, contributors, or algorithm versions.Collections would appear in user-specific and, eventually, group-specific catalogs. The same Collection could appear under both “Alice’s Collections” and “Boreal Biomass Team” without duplicating its Items or files. Appearing in a group catalog would not transfer ownership or grant permission to publish.
Metadata for Items and Collections
The following fields are proposed discovery metadata (in progress here: MAAP-Project/maap-dps-stac-extension#2). We should reuse existing STAC fields where they fit and reserve custom fields for DPS-specific information.
All of these fields are derived from the metdata files that are staged out with every DPS job (i.e. *.met.json, *.context.json0).
Items: describe each output
maap-dps:algorithm_nameesa-biomass-gamma0-fetch; the registered DPS algorithm that produced the output.processing:version1.0.0; the version of that same algorithm or processing chain.maap-dps:usernamealice; distinct from Collection ownership.maap-dps:tagexperiment-42, when supplied.processing:datetimederived_from), and an execution record (processing-execution) when available.Keep searchable metadata on every Item even when its Collection contains only one algorithm/version. This makes searches independent of Collection naming and grouping.
The Processing extension already defines
processing:version, but lacks a corresponding primary software/chain name. We are considering an upstream request forprocessing:name. Until such a field exists, retainmaap-dps:algorithm_name; do not use free-textprocessing:lineageas an identifier.Collections: describe the group of outputs
id,title,descriptionextent.spatial,extent.temporalsummaries["maap-dps:algorithm_name"]summaries["processing:version"]providers,license, relevant linksCollection summaries describe the values found across Items. They should reflect the contents rather than values inferred from the Collection ID. Job tags would remain Item-level metadata, not part of this Collection metadata proposal.
What users could find
experiment-42outputs from algorithm version1.0.0, within a region or time period, across Collections.Collection-level metadata would support Collection Search; Item properties would support targeted Item Search through the STAC API Filter extension (CQL2). Users would not need to know Collection IDs in advance.
One caveat: a Collection containing algorithm A/version 1 and algorithm B/version 2 summarizes both names and both versions. Those summaries alone cannot establish that it contains A/version 2. Exact algorithm/version matching in mixed Collections needs Item-level filtering, or future Collection search behavior that checks matching Items.
Related work
Richer Collection descriptions and searchable summaries depend on future improvements to the boilerplate Collection objects created by the STAC ingestion process. Those improvements can follow the initial change to tag-free Collection IDs and Item metadata.
The interfaces for creating Collections, selecting them during job submission, and managing ownership are a separate effort described in the DPS STAC integration planning documents. User catalogs are the initial browsing focus; group catalogs and group-owned Collections are part of the broader vision. Catalog browsing also does not automatically provide search scoped to a catalog.
Feedback requested
cc @sujen1412 @wildintellect