Skip to content

Error-handling sweep: crash/logic bug fixes, ERROR-token crash logging, shared helpers#39

Open
ggmarshall wants to merge 3 commits into
mainfrom
error-handling-sweep
Open

Error-handling sweep: crash/logic bug fixes, ERROR-token crash logging, shared helpers#39
ggmarshall wants to merge 3 commits into
mainfrom
error-handling-sweep

Conversation

@ggmarshall

Copy link
Copy Markdown
Contributor

Systematic bug-fix and error-handling sweep across the package, following the same review recently applied to legend-dataflow.

Crash bugs

  • par-geds-hit-aoe crashed with NameError on every run without --override-files; same undefined-in-branch class in hit/ecal.py (hit_dict/in_results_dict without --in-hit-dict) and dsp/eopt.py (optimiser plot objects when run_eopt: false).
  • hit/qc.py crashed for any channel with <4000 clean events (rng.choice(..., replace=False)).
  • hit/ecal.py guess-offset lookup subscripted a string default (.get(key, "fallback")["parameters"]) — the default is a fallback key into hit_dict.
  • build-tier-hit-single-channel passed an empty {} to build_hit and discarded the computed config (silently useless output); it now passes hit_config=hit_cfg and extracts ["pars"] like the multi-channel path.
  • --table-map (build-tier-dsp/hit) and --qbb-grid-path (par-geds-dsp-eopt) are now required=True — the rules always pass them and omitting them crashed with TypeError.
  • dataflow -v with no subcommand raised AttributeError; it now prints help and exits 1.
  • cfgtools.get_channel_config evaluated the __default__ lookup eagerly, raising KeyError even when the channel was present.
  • filedb.py ran scan_files twice, with the second call outside the error-context wrapper.

Wrong results

  • hit/ecal.py get_median/get_err had inverted conditions — they returned NaN for well-populated time bins and values for near-empty ones, effectively blanking pulser-stability monitoring.
  • alias_table tested struct membership by substring, so adding an alias whose name is a substring of an existing field (e.g. raw next to raw_blind) silently skipped the datatype registration.
  • as_ro on Path inputs returned only the rewritten basename, dropping all directory components.
  • hit/lq.py results are now stored per timestamp, mirroring hit/aoe.py.

Logging: crashes now reach log files as ERROR records

build_log's excepthook wrote raw tracebacks via traceback.print_exception to a handler stream discovered from the named logger — which has no handlers of its own in the dictConfig path (the FileHandler sits on root), so uncaught tracebacks fell back to stdout and never reached the per-job log file; even when they did, they carried no ERROR token for downstream log scanning. The excepthook now emits log.error("uncaught exception", exc_info=...) (formatted, severity-tagged, propagates to the root FileHandler), delegates KeyboardInterrupt to the default hook, and the fallback logger name is honoured in the dictConfig branch.

New shared helpers (legenddataflowscripts.utils)

expand_filelist, check_pulser_mask, require_config_keys, get_rule_config, and the fixed get_channel_config (now with an optional name= for error context). Adopted across the hit/dsp scripts: filelist arguments fail with named errors on empty/missing input, every mask[threshold_mask] is preceded by a length check that names the table and suggests mismatched pulser/input filelists, and the run_*/calibration config contracts are validated up front naming the channel and config file. These helpers are also intended to replace the duplicated copies in legend-dataflow (companion change there; its pin will need a bump to the next release).

Tests (18 → 49)

  • Console-script smoke test: imports every [project.scripts] entry point and runs --help (17 scripts) — catches broken imports, entry-point typos, and argparse regressions.
  • Unit tests for all new error paths: helper behaviors, the get_channel_config eager-default regression, as_ro Path regression, dataflow no-subcommand, alias_table membership on a synthetic HDF5 file, and the build_log excepthook (asserts the ERROR record + traceback lands in the log file).

Noted, not changed

  • eval() of config strings in aoe/lq/ecal/dplms/eopt — replacing these needs a whitelist built from the names production configs actually use; follow-up.
  • CLI contract note: aoe/lq/ecal previously treated files[0] as a filelist unconditionally; with expand_filelist a single argument is only expanded when it has a .filelist suffix (the Snakemake rules always pass .filelist paths).

Verification

  • pytest: 49 passed (warnings-as-errors)
  • ruff check / ruff format --check: clean
  • dataflow -v: prints help, exits 1, no traceback

Per AI_POLICY.md: this contribution was drafted with AI assistance (Claude Code) and reviewed by the submitter.

🤖 Generated with Claude Code

ggmarshall and others added 3 commits July 11, 2026 11:18
- build_log: route uncaught exceptions through log.error(exc_info=...) so
  crashes land in the log file as formatted ERROR records (the previous
  excepthook wrote raw tracebacks to a handler stream discovered from the
  named logger, which is empty in the dictConfig path -- tracebacks went to
  stdout, not the log). KeyboardInterrupt is delegated to the default hook,
  and the fallback logger name is now honoured in the dictConfig branch.
- cfgtools.get_channel_config: don't evaluate the __default__ lookup
  eagerly (raised KeyError even when the channel was present); named error
  when both channel and default are missing.
- new shared helpers: expand_filelist (utils/files.py), check_pulser_mask
  (pulser_removal), require_config_keys and get_rule_config (cfgtools),
  all exported from utils; get_pulser_mask names the file on a missing
  'mask' key.
- alias_table: test struct membership against the parsed field list
  instead of substring (adding 'raw' when 'raw_blind' exists was skipped);
  named error for non-struct parent datatypes.
- as_ro: keep directory components for Path inputs (previously returned
  only the rewritten basename).
- execenv: 'dataflow -v' (no subcommand) prints help and exits 1 instead
  of AttributeError; unknown --system names the config file and available
  systems.
- filedb: remove duplicated unguarded scan_files call; add missing
  f-prefix on the empty-file error (printed literal {row.raw_file}); named
  error for ignore-keys files without 'unprocessable'; drop dead
  exact-match ignore exemptions (rows are dropped by substring earlier).
- pyproject: pylint py-version 3.11 to match requires-python.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipts

Crash bugs:
- hit/aoe: override_dict was only defined when --override-files was given
  but passed unconditionally (NameError on every run without overrides).
- hit/ecal: hit_dict/in_results_dict undefined without --in-hit-dict;
  guess_offset_param default was subscripted as a dict (it is a fallback
  key into hit_dict).
- hit/qc: rng.choice(n, 4000, replace=False) crashed for channels with
  fewer than 4000 clean events (now min(4000, n)); overwrite loop crashed
  when the config has no fft_fields; parameters=None iterations.
- tier/hit single-channel: passed an empty dict to build_hit and discarded
  the computed config; now passes hit_config=hit_cfg and extracts ['pars']
  from channel entries like the multi-channel path.
- tier/dsp + tier/hit: --table-map is required (rules always pass it;
  omitting it crashed with TypeError on None).
- dsp/eopt: optimiser plots referenced bopt_*/sample_x when run_eopt was
  false (NameError); --qbb-grid-path is required (used unconditionally).
- dsp/nopt: dict(nopt_pars=..., **db_dict) raised TypeError when db_dict
  already contained nopt_pars.

Wrong results:
- hit/ecal get_median/get_err had inverted conditions: they returned NaN
  for well-populated bins and a value for near-empty ones, blanking the
  pulser-stability output.
- hit/lq: results are now stored per timestamp, mirroring aoe.

Error handling:
- adopt expand_filelist for the filelist idioms (empty/None inputs raise
  named errors), check_pulser_mask before every mask[threshold_mask]
  (length mismatch names the table and suggests mismatched filelists),
  and require_config_keys for the run_* and calibration config contracts
  (missing keys name the channel and config file).
- evtsel/pz: explicit 'either --pulser-file or --no-pulse is required'
  instead of Props.read_from(None).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- test_console_scripts.py: import every [project.scripts] entry point and
  run it with --help (catches broken imports, entry-point typos and
  argparse regressions for all 17 scripts).
- test_utils_helpers.py: get_channel_config (incl. the eager-default
  regression), expand_filelist, check_pulser_mask, require_config_keys,
  get_rule_config, get_pulser_mask missing-mask error, alias_table struct
  membership on a synthetic HDF5 file, and the build_log excepthook
  (ERROR record with traceback in the log file; KeyboardInterrupt
  delegated).
- test_util.py: as_ro str/Path/list cases (Path regression).
- test_execenv.py: dataflow with no subcommand exits 1 with usage;
  _select_execenv names the config file and available systems.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant