Repository navigation
Add chain_time_intervals to merge_index_files - #279
Merged
adperezm merged 1 commit intoOct 14, 2026
Merged
Conversation
Each index is written on its own, so its first entry's `time_previous_output` is the start time that index was given. Merging several indices into one time series leaves every index's first entry measured from a start time rather than from the entry before it, so the intervals overlap and grow. `average_field_files` weights every file by `time_interval`, so the late files then dominate the average. Add `chain_time_intervals`, which measures each interval from the preceding entry in time. Only entries that disagree with their predecessor are changed, so intervals that are already consistent are left alone. It is off by default, because merged indices are not always one series: for an ensemble each member is measured from its own start and chaining across members would be meaningless. When it is off and the inconsistency is present, log a warning, so a merged time series does not go unnoticed. It requires `sort_by_time`, since the intervals are chained along the sorted sequence. Also preserve `statistics_start_time` through the merge, which is what the first entry's interval is measured from, and make the sort stable so entries with the same time keep their indexed order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Another Claude-generated version of a fix I needed during postprocessing. I'm not sure if this is the best way to address the issue, or if there's another intended workflow, so let me know. In particular, this kinda assumes that you write out at time intervals coinciding with restarts, and you haven't accidentally lost a batch of statistics.
Ideally, Neko would write out the start/stop times or interval, but I guess the fld file format prohibits this...
Backstory:
I usually store my statistics batches in different folders to avoid accidentally overwriting them (perhaps not necessary, but suggested in the Neko user docs). Therefore, each index file is written on its own, so its first entry's
time_previous_outputis the start time that index file was given. Merging index files from each folder into one time series leaves every index's first entry measured from a start time rather than from the entry before it, so the recordedtime_intervalgrows and overlaps previous batches.This PR:
This PR adds
chain_time_intervals, which measures each interval from the preceding entry in time. The idea is to chain together statistics that were recorded continuously in time for the same run. Only entries that disagree with their predecessor are changed, so intervals that are already consistent are left alone. It is off by default, because merged indices are not always one series, e.g. for an ensemble. It requiressort_by_time, since the intervals are chained along the sorted sequence.