Get product versions from dependency json#3313
Open
vineetbansal wants to merge 6 commits into
Open
Conversation
Collaborator
Author
|
In case tests fail on python 3.10 on ubuntu - something funky is happening which is unrelated to this PR. This happens in an internal PR in my fork as well, where I've made no substantial change at all. I'm hoping that a change to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes issue #3266
Change Summary
The dependency string in
imap_cliis now assumed to supply the desired version major/minor for each product, for example:The keys in the
versionkey of the json are assumed to be the last part of theLogical_sourceof the xarray dataset (which has_separated strings). This is the same assumption as the current code, which usesLogical_sourceto determine the descriptor:cdf/utils.py-write_cdffunction:Note the following behavior that allows gradual adoption and testing.
--versionargument inimap_cliis still supported, but is optional.--versionargument is used to determine the descriptor versions of any descriptors that do not appear in the dependency string above. If the dependency string provides the version for a descriptor, it is used.--versionis used as a fallback for all descriptors.I'm hoping that 1+2+3 will allow testing of this code quickly with dagster on a small set of products to see if expected descriptors are generated with expected version numbers. I can take off one or more or all of these 3 behaviors, either in this PR, or in future PRs.
One assumption I'm making here:
Data_versionattribute of the CDF is populated to be exactly the version (for example,v017.0042). This is because something in the in-memory CDF needs to keep the version around as the dataset object is passed from one function to another, before it is written to disk. This could be any attribute really (or could be additional arguments we introduce between functions), butData_versionseems like the logical choice.However, if the one-off migration script that SDC has written populates this attribute a different way, then things will not be consistent between the one-off script and
imap_processing. If a different format ofData_versionis assumed by the migration script, then we need to make the corresponding change here.