Repository navigation
Fix multiple sample rates, skew and DAT reading issues - #53
Merged
Merged
Conversation
- Add pyproject.toml (hatchling) so the package is built from this repository; the version is read from comtrade.__version__ - Add numpy and pandas extras - Require Python >= 3.11 (the currently supported versions), bump to 0.2.0 - Rename tests/tests.py to tests/test_comtrade.py and run it with pytest - Replace the test workflow with CI: ruff, tests on Python 3.11-3.15 (plus Windows and macOS) and a build/twine check - Add a release workflow that publishes to PyPI via Trusted Publishing - Add Dependabot for GitHub Actions updates - Fix ruff findings (unused variables/import, type comparison) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPnfFrvYTuVEM9uDp81XiD
- Compute sample times across multiple sample rate segments: each segment continues from the last sample of the previous one, instead of (n - 1) / samp with the segment's own rate (see #45) - Accept a blank (non-critical) timestamp in ASCII DAT files and compute it from the sample rate (#48) - Fix loading binary DAT files with status channels only (#49) - Add Comtrade.analog_time with each analog channel's time plus its time skew (#29) - Read ASCII DAT files in blocks and unpack binary DAT fields for all rows at once (numpy is used when use_numpy_arrays is set), replacing the struct row-by-row reader - Update README supported features table and add tests Fixes #48, fixes #49, fixes #29 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VXtokXPfpX3vJri639eqXo
- Compute sample times across multiple sample rate segments: each segment continues from the last sample of the previous one, instead of (n - 1) / samp with the segment's own rate (see #45) - Accept a blank (non-critical) timestamp in ASCII DAT files and compute it from the sample rate (#48) - Fix loading binary DAT files with status channels only (#49) - Add Comtrade.analog_time with each analog channel's time plus its time skew (#29) - Update README supported features table and add tests Fixes #48, fixes #49, fixes #29 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VXtokXPfpX3vJri639eqXo
- Unpack each binary DAT field (sample number, timestamp, analog channel, 16-bit status group) for all rows at once with strided byte slices, replacing the struct row-by-row reader - Scale binary analog values and extract status bits with numpy when use_numpy_arrays is set - Read ASCII DAT files in blocks of lines (up to 1M values), converting each channel of a block at once - Binary DAT files ending in a partial row load their whole rows instead of failing with struct.error - Add tests for ASCII block reading, DAT files with more or fewer rows than the CFG samples and numpy arrays reading Loading the issue's sample (ASCII, 2212 analog channels, 10000 samples) drops from 3.6 s to 2.0 s. Binary files with 2212 channels load about 2x faster, and 5.5 to 10x faster with use_numpy_arrays. Fixes #24 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XUdNv59jhgaaRKFfZmRMwQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(n - 1) / sampwith only the current segment's rate, so every sample after a rate change got a wrong time when the DAT timestamps were missing or non-critical.Comtrade.analog_timeproperty with each analog channel's time plus its skew (µs, from the start of the sample period). It is computed the first time it's accessed.Comtrade.timeandto_dataframe()still use the sample time.use_numpy_arrays=True). This replaces the row-by-rowstructreader.analog_timeis documented.Tests
New tests cover two sample rates (ASCII and binary), blank timestamps, status-only binary files (
BINARY,BINARY32,FLOAT32), skewed channels (ASCII, binary, numpy), block reading, DAT files with more or fewer rows than the CFG says, and numpy versusarray.arrayresults. Locally, 150 tests pass andruff checkis clean.Note
This branch is based on #51 (packaging and CI), so this PR also shows that commit until #51 is merged.
Fixes #48
Fixes #49
🤖 Generated with Claude Code
https://claude.ai/code/session_01UHqqatRKrragoYDgQKzpmh