Skip to content

Get product versions from dependency json#3313

Open
vineetbansal wants to merge 6 commits into
IMAP-Science-Operations-Center:devfrom
vineetbansal:vb/issue3266
Open

Get product versions from dependency json#3313
vineetbansal wants to merge 6 commits into
IMAP-Science-Operations-Center:devfrom
vineetbansal:vb/issue3266

Conversation

@vineetbansal

@vineetbansal vineetbansal commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Closes issue #3266

Change Summary

The dependency string in imap_cli is now assumed to supply the desired version major/minor for each product, for example:

{
            "dependency": [
                {
                    "type": "ancillary",
                    "files": [
                        "imap_mag_l1b-cal_20250101_v001.cdf",
                        "imap_mag_l1b-cal_20250103_20250104_v002.cdf"
                    ]
                },
                {
                    "type": "science",
                    "files": [
                        "imap_idex_l2_sci_20240312_v000.cdf",
                        "imap_idex_l2_sci_20240312_v001.cdf"
                    ]
                }
            ],
            "version": {
                "cal": {"major_version": 2, "minor_version": 1}
            }
        }

The keys in the version key of the json are assumed to be the last part of the Logical_source of the xarray dataset (which has _ separated strings). This is the same assumption as the current code, which uses Logical_source to determine the descriptor:

cdf/utils.py - write_cdf function:

instrument, data_level, descriptor = dataset.attrs["Logical_source"].split("_")[1:]

Note the following behavior that allows gradual adoption and testing.

  1. The --version argument in imap_cli is still supported, but is optional.
  2. If provided, the --version argument is used to determine the descriptor versions of any descriptors that do not appear in the dependency string above. If the dependency string provides the version for a descriptor, it is used.
  3. The old format of the dependency string is still supported - in that case, the version provided by --version is used as a fallback for all descriptors.

I'm hoping that 1+2+3 will allow testing of this code quickly with dagster on a small set of products to see if expected descriptors are generated with expected version numbers. I can take off one or more or all of these 3 behaviors, either in this PR, or in future PRs.

One assumption I'm making here:

  • The Data_version attribute of the CDF is populated to be exactly the version (for example, v017.0042). This is because something in the in-memory CDF needs to keep the version around as the dataset object is passed from one function to another, before it is written to disk. This could be any attribute really (or could be additional arguments we introduce between functions), but Data_version seems like the logical choice.

However, if the one-off migration script that SDC has written populates this attribute a different way, then things will not be consistent between the one-off script and imap_processing. If a different format of Data_version is assumed by the migration script, then we need to make the corresponding change here.

@vineetbansal vineetbansal changed the title Vb/issue3266 Get product versions from dependency json Jun 19, 2026
@vineetbansal vineetbansal changed the title Get product versions from dependency json WIP: Get product versions from dependency json Jun 19, 2026
@vineetbansal vineetbansal changed the title WIP: Get product versions from dependency json Get product versions from dependency json Jun 19, 2026
@vineetbansal

Copy link
Copy Markdown
Collaborator Author

In case tests fail on python 3.10 on ubuntu - something funky is happening which is unrelated to this PR. This happens in an internal PR in my fork as well, where I've made no substantial change at all. I'm hoping that a change to dev and a merge here will fix it. This can be done if I'm not around in the next few days and things need to move fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant