-
Notifications
You must be signed in to change notification settings - Fork 28
initializing spectroscopy #196
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
smribet
wants to merge
213
commits into
dev
Choose a base branch
from
spectroscopy
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+7,736
−8
Open
Changes from all commits
Commits
Show all changes
213 commits
Select commit
Hold shift + click to select a range
1038d20
starting spectroscopy class
smribet 9a67b3a
moving dataset3dspectroscopy
smribet cb2f678
updating inits for file loading
smribet f87fb21
Swapping to (E,x,y) at least for now. We may want validators.
smribet 51fe26b
adding EDS and EELS
smribet 32e8218
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 09fe109
test commit
Sangoda e4c9b3b
checking pull
smribet f697492
test
4329f14
pull test
smribet 57e0b70
DataSpectroscopy class for EELS and EDS.
Sangoda 760ff16
added pca and plot_virtual_image
f5a0635
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet b73cbb8
fixes
5e46224
file_readers fix for 3d spectroscopy data
Sangoda bd78da1
Updated the show_mean_spectrum function
Sangoda 97830e7
added comment line as test commit
nwvlahakis 447ce32
removed test comment line
nwvlahakis 518f4ee
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet a54b63a
adding metadata to spectroscopy
smribet 7c9aa10
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet fbf8900
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 3b2f977
adding subtract_background function and moving background subtraction…
nwvlahakis c8dea15
Merge remote-tracking branch 'upstream/spectroscopy' into spectroscopy
nwvlahakis 7bfa0a9
removing redundant background subtraction function definitions from d…
nwvlahakis fa1eaba
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet edba0b8
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet 0b2d5c7
first pass at torch model for EDS
smribet ddb2e2f
models
smribet 96f975f
better plotting
smribet c7aa242
more flexible FWHM. let's see if this helps.
smribet e3b135a
add energy_range and cleanup background subtraction functions
nwvlahakis 7422f05
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet ca07d1d
fix transpose bug for digitalmicrograph EELS data in read_3d_spectros…
nwvlahakis 2c8b5ec
modify print message to inform where 3D datasets are located in case …
nwvlahakis 510a3a7
small edits for precommit compliance in dataset3dspectroscopy
nwvlahakis 834792d
add checks/error messages for out-of-bounds energy range inputs
nwvlahakis 44658b6
flags for returning back-subtracted 3D datasets and storage for back-…
nwvlahakis 7db326e
edits to spectrum data storage flags/structure
nwvlahakis 1f6bbb3
axis label fixes for EELS
nwvlahakis 85a2117
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis f19cb55
improve optimizer
smribet 48905a1
move to 3D fitting
smribet b222257
make plotting better
smribet 5f7d8dc
change loss
smribet 692b0db
reformatting, adding EELS
smribet 89b93ce
lr flexibility
smribet 69006b1
normalization fixes
smribet b31ee7c
cleaning up
smribet ff2cae0
plotting update
smribet fffe363
make gpu compatible
smribet aec39c6
a few more changes
smribet 75e8adc
Apply local spectroscopy edits after pulling latest
Sangoda 420e466
quantem spectroscopy: update dataset3dspectroscopy and add atomic wei…
Sangoda 3029623
updates to zlp
64f8356
Merge branch 'spectroscopy' of https://github.com/electronmicroscopy/…
0959777
small cleanup and eels axis label fix
nwvlahakis c63b375
Merge remote-tracking branch 'upstream/spectroscopy' into spectroscopy
nwvlahakis f6a3466
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 8241123
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet fa5a58d
dataset1d for spectra
nwvlahakis afb05f0
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis d1e1d7f
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet f935bb2
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet 422da10
removing titan k factors
smribet c69cb66
removing titan k factors
smribet 88dafe2
Added a dedicated EDS auto-ID flow (peak_autoid).
Sangoda 1800cf6
Merge branch 'spectroscopy' of https://github.com/electronmicroscopy/…
Sangoda 0822824
-Added a new auto-ID method for EDS peaks
Sangoda 266dc74
let's see if I broke the repo... changing X-ray lines
smribet 746d5ca
patch of fitting
smribet 7978c2b
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis 88e9cac
1d spectra attached to 3d spectroscopy datasets now saved as dataset1…
nwvlahakis 8a6325a
cleanup
nwvlahakis af5ec6e
simplifying
smribet 5dfbb7d
a lot simplified
smribet c8f5865
doc strings
smribet 0c17bd4
atomics weights update
smribet a3f1461
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 3c1990a
restructure
smribet a9b0766
remove old code
smribet 8bf94e0
lookup energy lines
smribet eeeba25
generate spectrum Images
smribet 8e643d3
plot spectrum images
smribet 5be8301
plotting update
smribet 6fc62b6
fix
smribet 522f50d
spectrum images pytorch
smribet 735d6e9
improve naming
smribet 181eecd
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet d23c9eb
update plotting
smribet 1b7ba7b
using model elements
smribet c9fab6d
refactor
smribet 9d280f7
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet e1f7bf5
trying to organize text parsing
smribet 59322df
CL adding table
smribet c867952
clear spectrum images, update history size for lbfgs
smribet 96927e5
add
smribet dbfb639
cleaning up show_mean_spectrum
smribet 32aaf64
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 72553fe
first chunk of powerlaw bg fit for eels and modified function call in…
nwvlahakis 1726386
user specified pre-edge window size option for eels bg fit (powerlaw)
nwvlahakis cc970ff
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 3463679
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet f53f0f9
fixing background curve fitting/plotting for eels
nwvlahakis 01db033
increase max iterations for curve fit
nwvlahakis 3e8b72d
draft EELS pca denoising function
nwvlahakis e4f1478
make pca denoising user energy_range compatible
nwvlahakis e3cd827
some cleanup
nwvlahakis 8edf922
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis 2f5633e
rolling average smoothing function for eels
nwvlahakis 2ee116d
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet a3d1746
remove sklearn and replace with pytorch pca
smribet 25d3dc0
remove pca redundancy, 3d dataset output from perform_pca, and cleanup
nwvlahakis 3c65fdb
remove smooth_eels_pca
nwvlahakis cc00c10
test
Sangoda 0b69231
Local changes to uv.lock
81f0f25
Ignore local data and notebooks
d300e16
Add thickness module and supporting alignment/visualization modules w…
ccf1517
build: update uv.lock to match pyproject.toml
9c67621
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 0e0b524
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet 4e9d9e1
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis baabca0
added modes to autoid, made changes to scoring for more accuracy (wip)
Sangoda 63898d2
Merge branch 'spectroscopy' of https://github.com/electronmicroscopy/…
Sangoda fccb83b
fixing mean_spectrum
smribet afd8dce
porting alignment/thickness functions to 3deels, removing widget and …
nwvlahakis b23d255
Update spectroscopy init by removing imports of deleted modules.
3c2e593
Added EELS visualization functions to monitor ZLP calibratrion, align…
14888e1
tab correct
96616e8
Double plotting corection in plot dual eels picker function.
5e286f3
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis 457933f
fix for spec file readers
smribet 7803d74
fix for load element info
smribet 16802bf
fix for roi and auto_id
Sangoda 795c87b
added docstrings to functions in dataset3deds.py
Sangoda f7bf217
auto_id labeling changes
Sangoda 9b9f019
bug fix
smribet 572044c
update return map logic
smribet 619071c
convert eV to keV in read_3d_spectroscopy; refactor model_elements ch…
Sangoda cf19ca3
quick fix
smribet c861012
Merge remote-tracking branch 'upstream' into spectroscopy
nwvlahakis 71edf75
updating xray csv
smribet 3a33280
_normalize_element_info
smribet 4bb4b13
Merge remote-tracking branch 'upstream/spectroscopy' into spectroscopy
nwvlahakis 00d2052
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis f50f5ed
remove redundant functions
nwvlahakis 652e862
beginning refactor of zlp code to separate "measure" and "correct" fu…
nwvlahakis 9bf4680
bug fixes
smribet a521ee6
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet 909aded
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 4f50f9e
some plotting and bug fixes
smribet b05c57b
changing data_type to self.dataset_type
smribet 48c980f
bug fix for background subtraction
smribet c8ee683
pca plotting
smribet 1507702
small fix
smribet 8c3a4cc
adding local EELS background fitting
smribet 108e16b
measure zlp refactor changes, a couple bugs to fix
nwvlahakis c428e3a
Merge remote-tracking branch 'upstream/spectroscopy' into spectroscopy
nwvlahakis 4b9557f
zlp 2d plane fit
nwvlahakis 01e67ba
user flag to make plane fit optional
nwvlahakis f98ac75
apply zlp correction function
nwvlahakis 0e8f97e
add plotting
nwvlahakis 494ad09
optional flags for measure and apply zlp functions
nwvlahakis 74a49bd
updating eels edges
smribet d8b0683
updating eels edges pt 2
smribet dacdfc1
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis 4553f40
fixes to plotting/cleanup for zlp alignment
nwvlahakis fb99b76
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet bcfe64c
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet 393d930
sanya's eds changes
smribet 9d07ae1
sanya's spectroscopy changes
smribet af85309
fixing labels
smribet 7f136bf
updated background subtraction EDS
smribet 9cf731b
bug fix
smribet e4ae004
normalization bug
smribet aca05b9
removing unused code
smribet 2f556e2
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 248fbb3
changes to fit for zlp
smribet d5cfaca
E,x,y to x,y,E refactor
nwvlahakis 788b488
spectrum images bug fix
nwvlahakis 5836c2d
fix for mean spectrum
smribet a5ce57f
other cleaning up
smribet dd164b5
Realtive Thicknesss Map was verticallty fliiped compared to othe maps…
840331f
Add return_shifts parameter to apply_zlp_correction for dual-EELS sup…
f500013
Add subtract_background_limited_preedge method for limited pre-edge data
af958c2
Merge remote-tracking branch 'origin/dev' into spectroscopy
smribet 7f234cf
change to row, column indexing
smribet c0c06b7
changing to XEDS
smribet 72a5444
removing kwargs
smribet 02f753d
updating read 4d and 3d to print dataset shapes
smribet 19b495f
removing excess floats and changing to lower case
smribet a22ae52
typo correct
smribet 3dfeffb
separating out plotting
smribet 8236365
peak autoid refactor
smribet 9155160
one more small change
smribet 0499af8
adding citations and separating out fitting
smribet 677bd22
small cleanup, underscore fixes
nwvlahakis 50e0fb6
user-input value fixes for eels calculate_background_iterative
nwvlahakis 8b9544e
plot_attached_spectrum bug fix
nwvlahakis adca338
dataset1d docstring fixes
nwvlahakis c6ce12f
pca bug fix
nwvlahakis 613447d
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis 595e46b
changing some returns from EDS
smribet 1780108
Merge remote-tracking branch 'origin/spectroscopy' into spectroscopy
smribet 7d7f18c
more fixes
smribet 65032b6
update to autoid
smribet 635203c
Merge remote-tracking branch 'upstream/dev' into spectroscopy
nwvlahakis 61fa162
delete redundant zlp offset measurement function
nwvlahakis 9d82d80
polynomial fit option for ZLP alignment
nwvlahakis bf3797b
apply_zlp_correction fix to allow polynomial fit
nwvlahakis b1a64dd
modify in place option for zlp correction
nwvlahakis 37cb4c1
update correct_zlp_shift to correct_high_loss_energy_axis
nwvlahakis 53e1ac6
fix to thickness mapping/log ratio function
nwvlahakis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,213 @@ | ||
| from typing import Self | ||
|
|
||
| import matplotlib.pyplot as plt | ||
| import numpy as np | ||
| from numpy.typing import NDArray | ||
|
|
||
| from quantem.core.datastructures.dataset import Dataset | ||
| from quantem.core.utils.validators import ensure_valid_array | ||
|
|
||
|
|
||
| @Dataset.register_dimension(1) | ||
| class Dataset1d(Dataset): | ||
| """1D dataset class that inherits from Dataset. | ||
|
|
||
| This class represents a 1D dataset, such as spectra from XEDS or EELS. | ||
|
|
||
| Attributes | ||
| ---------- | ||
| None beyond base Dataset. | ||
| """ | ||
|
|
||
| def __init__( | ||
| self, | ||
| array: NDArray, | ||
| name: str, | ||
| origin: NDArray | tuple | list | float | int, | ||
| sampling: NDArray | tuple | list | float | int, | ||
| units: list[str] | tuple | list | None = None, | ||
| signal_units: str = "arb. units", | ||
| metadata: dict = {}, | ||
| _token: object | None = None, | ||
| ): | ||
| """Initialize a 1D dataset. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| array : NDArray | ||
| The underlying 1D array data | ||
| name : str | ||
| A descriptive name for the dataset | ||
| origin : NDArray | tuple | list | float | int | ||
| The origin coordinates for each dimension | ||
| sampling : NDArray | tuple | list | float | int | ||
| The sampling rate/spacing for each dimension | ||
| units : list[str] | tuple | list | ||
| Units for each dimension | ||
| signal_units : str, optional | ||
| Units for the array values, by default "arb. units" | ||
| _token : object | None, optional | ||
| Token to prevent direct instantiation, by default None | ||
| """ | ||
| super().__init__( | ||
| array=array, | ||
| name=name, | ||
| origin=origin, | ||
| sampling=sampling, | ||
| units=units, | ||
| signal_units=signal_units, | ||
| metadata=metadata, | ||
| _token=_token, | ||
| ) | ||
|
|
||
| @classmethod | ||
| def from_array( | ||
| cls, | ||
| array: NDArray, | ||
| name: str | None = None, | ||
| origin: NDArray | tuple | list | float | int | None = None, | ||
| sampling: NDArray | tuple | list | float | int | None = None, | ||
| units: list[str] | tuple | list | None = None, | ||
| signal_units: str = "arb. units", | ||
| ) -> Self: | ||
| """Create a Dataset1d from a 1D array. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| array : NDArray | ||
| 1D array with shape (length) | ||
| name : str | None | ||
| Dataset name. Default: "1D dataset" | ||
| origin : NDArray | tuple | list | float | int | None | ||
| Origin for each dimension. Default: [0] | ||
| sampling : NDArray | tuple | list | float | int | None | ||
| Sampling for each dimension. Default: [1] | ||
| units : list[str] | tuple | list | None | ||
| Units for each dimension. Default: ["pixels"] | ||
| signal_units : str | ||
| Units for array values. Default: "arb. units" | ||
|
|
||
| Returns | ||
| ------- | ||
| Dataset1d | ||
|
|
||
| Examples | ||
| -------- | ||
| >>> import numpy as np | ||
| >>> from quantem.core.datastructures import Dataset1d | ||
| >>> arr = np.random.rand(10) | ||
| >>> data = Dataset1d.from_array(arr) | ||
| >>> data.shape | ||
| (10,) | ||
|
|
||
| With calibration: | ||
|
|
||
| >>> data = Dataset1d.from_array( | ||
| ... arr, | ||
| ... sampling=[0.15], | ||
| ... units=["eV"], | ||
| ... ) | ||
|
|
||
| Visualize: | ||
|
|
||
| >>> data.show() # all frames in grid | ||
| >>> data.show(index=0) # single frame | ||
| >>> data.show(ncols=2) # 2 columns | ||
| """ | ||
| array = ensure_valid_array(array, ndim=1) | ||
| return cls( | ||
| array=array, | ||
| name=name if name is not None else "1D dataset", | ||
| origin=origin if origin is not None else np.zeros(1), | ||
| sampling=sampling if sampling is not None else np.ones(1), | ||
| units=units if units is not None else ["pixels"], | ||
| signal_units=signal_units, | ||
| _token=cls._token, | ||
| ) | ||
|
|
||
| @classmethod | ||
| def from_shape( | ||
| cls, | ||
| shape: tuple[int], | ||
| name: str = "constant 1D dataset", | ||
| fill_value: float = 0.0, | ||
| origin: NDArray | tuple | list | float | int | None = None, | ||
| sampling: NDArray | tuple | list | float | int | None = None, | ||
| units: list[str] | tuple | list | None = None, | ||
| signal_units: str = "arb. units", | ||
| ) -> Self: | ||
| """Create a Dataset1d filled with a constant value. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| shape : tuple[int] | ||
| Shape (length) | ||
| name : str | ||
| Dataset name. Default: "constant 1D dataset" | ||
| fill_value : float | ||
| Value to fill array with. Default: 0.0 | ||
| origin : NDArray | tuple | list | float | int | None | ||
| Origin for each dimension | ||
| sampling : NDArray | tuple | list | float | int | None | ||
| Sampling for each dimension | ||
| units : list[str] | tuple | list | None | ||
| Units for each dimension | ||
| signal_units : str | ||
| Units for array values | ||
|
|
||
| Returns | ||
| ------- | ||
| Dataset1d | ||
|
|
||
| Examples | ||
| -------- | ||
| >>> data = Dataset1d.from_shape((10000)) | ||
| >>> data.shape | ||
| (10000,) | ||
| >>> data.array.max() | ||
| 0.0 | ||
| """ | ||
| array = np.full(shape, fill_value, dtype=np.float32) | ||
| return cls.from_array( | ||
| array=array, | ||
| name=name, | ||
| origin=origin, | ||
| sampling=sampling, | ||
| units=units, | ||
| signal_units=signal_units, | ||
| ) | ||
|
|
||
| def show( | ||
| self, | ||
| title: str | None = None, | ||
| returnfig: bool = False, | ||
| ): | ||
| """ | ||
| Plots 1D dataset | ||
|
|
||
| Parameters | ||
| ---------- | ||
| title: str | ||
| Title of Dataset | ||
| returnfig: bool | ||
| Option to include figure as return value for method | ||
| """ | ||
|
|
||
| if title is None: | ||
| title = self.name | ||
|
|
||
| fig, (ax) = plt.subplots(1, 1, figsize=(4, 4)) | ||
|
|
||
| ax.plot( | ||
| float(self.origin[0]) + float(self.sampling[0]) * np.arange(self.shape[0]), | ||
| self.array, | ||
| linewidth=1.5, | ||
| ) | ||
| ax.set_xlabel(self.units[0]) | ||
| ax.set_ylabel(self.signal_units) | ||
| ax.set_title(title) | ||
|
|
||
| fig.tight_layout() | ||
| plt.show() | ||
|
|
||
| return (fig, ax) if returnfig else None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could add a function to print out the dataset indices, shapes, and other metadata line names if available.