Error-handling sweep: crash/logic bug fixes, ERROR-token crash logging, shared helpers#39
Open
ggmarshall wants to merge 3 commits into
Open
Error-handling sweep: crash/logic bug fixes, ERROR-token crash logging, shared helpers#39ggmarshall wants to merge 3 commits into
ggmarshall wants to merge 3 commits into
Conversation
- build_log: route uncaught exceptions through log.error(exc_info=...) so
crashes land in the log file as formatted ERROR records (the previous
excepthook wrote raw tracebacks to a handler stream discovered from the
named logger, which is empty in the dictConfig path -- tracebacks went to
stdout, not the log). KeyboardInterrupt is delegated to the default hook,
and the fallback logger name is now honoured in the dictConfig branch.
- cfgtools.get_channel_config: don't evaluate the __default__ lookup
eagerly (raised KeyError even when the channel was present); named error
when both channel and default are missing.
- new shared helpers: expand_filelist (utils/files.py), check_pulser_mask
(pulser_removal), require_config_keys and get_rule_config (cfgtools),
all exported from utils; get_pulser_mask names the file on a missing
'mask' key.
- alias_table: test struct membership against the parsed field list
instead of substring (adding 'raw' when 'raw_blind' exists was skipped);
named error for non-struct parent datatypes.
- as_ro: keep directory components for Path inputs (previously returned
only the rewritten basename).
- execenv: 'dataflow -v' (no subcommand) prints help and exits 1 instead
of AttributeError; unknown --system names the config file and available
systems.
- filedb: remove duplicated unguarded scan_files call; add missing
f-prefix on the empty-file error (printed literal {row.raw_file}); named
error for ignore-keys files without 'unprocessable'; drop dead
exact-match ignore exemptions (rows are dropped by substring earlier).
- pyproject: pylint py-version 3.11 to match requires-python.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipts Crash bugs: - hit/aoe: override_dict was only defined when --override-files was given but passed unconditionally (NameError on every run without overrides). - hit/ecal: hit_dict/in_results_dict undefined without --in-hit-dict; guess_offset_param default was subscripted as a dict (it is a fallback key into hit_dict). - hit/qc: rng.choice(n, 4000, replace=False) crashed for channels with fewer than 4000 clean events (now min(4000, n)); overwrite loop crashed when the config has no fft_fields; parameters=None iterations. - tier/hit single-channel: passed an empty dict to build_hit and discarded the computed config; now passes hit_config=hit_cfg and extracts ['pars'] from channel entries like the multi-channel path. - tier/dsp + tier/hit: --table-map is required (rules always pass it; omitting it crashed with TypeError on None). - dsp/eopt: optimiser plots referenced bopt_*/sample_x when run_eopt was false (NameError); --qbb-grid-path is required (used unconditionally). - dsp/nopt: dict(nopt_pars=..., **db_dict) raised TypeError when db_dict already contained nopt_pars. Wrong results: - hit/ecal get_median/get_err had inverted conditions: they returned NaN for well-populated bins and a value for near-empty ones, blanking the pulser-stability output. - hit/lq: results are now stored per timestamp, mirroring aoe. Error handling: - adopt expand_filelist for the filelist idioms (empty/None inputs raise named errors), check_pulser_mask before every mask[threshold_mask] (length mismatch names the table and suggests mismatched filelists), and require_config_keys for the run_* and calibration config contracts (missing keys name the channel and config file). - evtsel/pz: explicit 'either --pulser-file or --no-pulse is required' instead of Props.read_from(None). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- test_console_scripts.py: import every [project.scripts] entry point and run it with --help (catches broken imports, entry-point typos and argparse regressions for all 17 scripts). - test_utils_helpers.py: get_channel_config (incl. the eager-default regression), expand_filelist, check_pulser_mask, require_config_keys, get_rule_config, get_pulser_mask missing-mask error, alias_table struct membership on a synthetic HDF5 file, and the build_log excepthook (ERROR record with traceback in the log file; KeyboardInterrupt delegated). - test_util.py: as_ro str/Path/list cases (Path regression). - test_execenv.py: dataflow with no subcommand exits 1 with usage; _select_execenv names the config file and available systems. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Systematic bug-fix and error-handling sweep across the package, following the same review recently applied to legend-dataflow.
Crash bugs
par-geds-hit-aoecrashed withNameErroron every run without--override-files; same undefined-in-branch class inhit/ecal.py(hit_dict/in_results_dictwithout--in-hit-dict) anddsp/eopt.py(optimiser plot objects whenrun_eopt: false).hit/qc.pycrashed for any channel with <4000 clean events (rng.choice(..., replace=False)).hit/ecal.pyguess-offset lookup subscripted a string default (.get(key, "fallback")["parameters"]) — the default is a fallback key intohit_dict.build-tier-hit-single-channelpassed an empty{}tobuild_hitand discarded the computed config (silently useless output); it now passeshit_config=hit_cfgand extracts["pars"]like the multi-channel path.--table-map(build-tier-dsp/hit) and--qbb-grid-path(par-geds-dsp-eopt) are nowrequired=True— the rules always pass them and omitting them crashed withTypeError.dataflow -vwith no subcommand raisedAttributeError; it now prints help and exits 1.cfgtools.get_channel_configevaluated the__default__lookup eagerly, raisingKeyErroreven when the channel was present.filedb.pyranscan_filestwice, with the second call outside the error-context wrapper.Wrong results
hit/ecal.pyget_median/get_errhad inverted conditions — they returned NaN for well-populated time bins and values for near-empty ones, effectively blanking pulser-stability monitoring.alias_tabletested struct membership by substring, so adding an alias whose name is a substring of an existing field (e.g.rawnext toraw_blind) silently skipped thedatatyperegistration.as_roonPathinputs returned only the rewritten basename, dropping all directory components.hit/lq.pyresults are now stored per timestamp, mirroringhit/aoe.py.Logging: crashes now reach log files as ERROR records
build_log's excepthook wrote raw tracebacks viatraceback.print_exceptionto a handler stream discovered from the named logger — which has no handlers of its own in the dictConfig path (the FileHandler sits on root), so uncaught tracebacks fell back to stdout and never reached the per-job log file; even when they did, they carried noERRORtoken for downstream log scanning. The excepthook now emitslog.error("uncaught exception", exc_info=...)(formatted, severity-tagged, propagates to the root FileHandler), delegatesKeyboardInterruptto the default hook, and thefallbacklogger name is honoured in the dictConfig branch.New shared helpers (
legenddataflowscripts.utils)expand_filelist,check_pulser_mask,require_config_keys,get_rule_config, and the fixedget_channel_config(now with an optionalname=for error context). Adopted across the hit/dsp scripts: filelist arguments fail with named errors on empty/missing input, everymask[threshold_mask]is preceded by a length check that names the table and suggests mismatched pulser/input filelists, and therun_*/calibration config contracts are validated up front naming the channel and config file. These helpers are also intended to replace the duplicated copies in legend-dataflow (companion change there; its pin will need a bump to the next release).Tests (18 → 49)
[project.scripts]entry point and runs--help(17 scripts) — catches broken imports, entry-point typos, and argparse regressions.get_channel_configeager-default regression,as_roPath regression,dataflowno-subcommand,alias_tablemembership on a synthetic HDF5 file, and thebuild_logexcepthook (asserts the ERROR record + traceback lands in the log file).Noted, not changed
eval()of config strings in aoe/lq/ecal/dplms/eopt — replacing these needs a whitelist built from the names production configs actually use; follow-up.files[0]as a filelist unconditionally; withexpand_filelista single argument is only expanded when it has a.filelistsuffix (the Snakemake rules always pass.filelistpaths).Verification
pytest: 49 passed (warnings-as-errors)ruff check/ruff format --check: cleandataflow -v: prints help, exits 1, no tracebackPer
AI_POLICY.md: this contribution was drafted with AI assistance (Claude Code) and reviewed by the submitter.🤖 Generated with Claude Code