Fix EPG scheduler storm, dead image host, add plugin.json - #4
Open
kilo-WATT wants to merge 2 commits into
Open
Conversation
Two bugs in plugin.py compound to make the auto-refresh scheduler
effectively broken from the first install:
1. _create_or_update_epg_source() looked up its own PluginConfig row
with `__name__.split(".")[0]`. Under Dispatcharr's real plugin
loader the module's __name__ is "_dispatcharr_plugin_zap2xml.plugin",
so this evaluated to "_dispatcharr_plugin_zap2xml" instead of the
actual config key "zap2xml" (see _plugin_key()). The lookup always
returned nothing, so `last_generated` was never persisted, even on
a fully successful run.
2. Because `last_generated` was never set, _scheduler_loop's should_run
check always concluded a refresh was overdue. Combined with the
fact that Dispatcharr runs multiple worker processes (gunicorn +
celery), each with its own independent scheduler thread and no
cross-process coordination, every worker fired download_epg on
every ~60s tick, forever. Concurrent runs then raced on the shared
workspace/output file (one run's cleanup step could delete the
produced XML out from under another run's copy-to-/data/epgs step),
so the copy silently failed and EPG data quietly went stale while
Gracenote got hammered continuously.
Fixes:
- Reuse the existing _plugin_key() helper instead of the broken
__name__-based lookup, so last_generated actually gets recorded.
- Add a flock()-based cross-process lock (_SCHED_LOCK_PATH) around
the scheduler's run, so only one worker process executes
download_epg at a time, with a re-check of "still due" after
acquiring the lock to avoid a redundant back-to-back run.
Verified end-to-end against a live Dispatcharr instance: prior to
the fix, last_generated was never set and the scheduler fired every
~60s indefinitely, hammering Gracenote's API and never actually
completing the copy+ingest step reliably. After the fix, exactly one
tick fires per interval, last_generated is correctly persisted, and
guide data ingests successfully into the EPG source.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Folded into this PR rather than a separate one, since both bugs were found while debugging the same broken guide-data pipeline: - zap2xml.py: swap the two hardcoded dshm.tmsimg.com references to demo.tmsimg.com (dshm no longer resolves — NXDOMAIN; demo serves the identical /assets/pXXXX_*.jpg paths, verified HTTP 200). - Add plugin.json (generated from the plugin's own fields/actions), clearing Dispatcharr's "legacy plugin" warning. - Bump plugin.json's version to 2.1.5 to match plugin.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3
Summary
Two independent bugs found while debugging why a live Dispatcharr instance's Jellyfin Live TV guide had stopped getting fresh data and thumbnails, both fixed in
plugin.py/zap2xml.py.1. EPG auto-refresh scheduler storm —
last_generatednever persists_create_or_update_epg_source()looks up its own settings row with:Under Dispatcharr's real plugin loader, this module's
__name__is_dispatcharr_plugin_zap2xml.plugin, soplugin_keyevaluates to_dispatcharr_plugin_zap2xml— notzap2xml, the actualPluginConfig.key(see the existing_plugin_key()helper, which gets this right and is already used elsewhere). The lookup silently returnsNoneevery time, solast_generatedis never written, even after a fully successful fetch + parse.Because
last_generatednever persists,_scheduler_loop'sshould_runcheck always evaluates true. Dispatcharr runs multiple worker processes (gunicorn + celery), each with its own independent scheduler thread and no cross-process coordination (_SCHEDULER_LOCKonly dedupes within one process), so every worker firesdownload_epgon every ~60s tick, forever. Concurrent runs then race on the shared workspace/output file — one run's cleanup step can delete the produced XML out from under another run's copy-to-/data/epgsstep, so the copy silently fails (caught by a bareexcept Exception, never logged), theEPGSourcenever gets refreshed, and the guide quietly runs out of programme data days later, while Gracenote's API gets hammered roughly once a minute in the meantime.Fix: reuse
_plugin_key()instead of the broken__name__-based lookup, and add aflock()-based cross-process lock (_SCHED_LOCK_PATH) around the scheduler's run, so only one worker process across the deployment ever executesdownload_epgat a time (with a re-check of "still due" after acquiring the lock, to skip a redundant back-to-back run at the interval boundary).2. Dead
dshm.tmsimg.comimage host (fixes #3)Gracenote retired the
dshm.tmsimg.comimage CDN subdomain (NXDOMAIN, confirmed against1.1.1.1), so every<programme><icon>in the generated XMLTV points at a dead host — Jellyfin logs thousands ofUnable to pre-cachewarnings per guide refresh and no thumbnails load. This is a recurring problem — the code already migrated once before (zap2it.tmsimg.com→dshm.tmsimg.com).Fix: swap the two hardcoded
dshm.tmsimg.comreferences (~line 481, ~line 499) todemo.tmsimg.com, which resolves and serves the identical/assets/pXXXX_*.jpgpaths (verifiedcurl -I→200 OK).3. Add
plugin.jsonmanifestThe plugin ships with no manifest, so Dispatcharr flags it as legacy ("Please update or ask the developer to add plugin.json"). Added one generated from the plugin's own
fields/actions, which clears the warning. Bumpedplugin.py's version 2.1.4 → 2.1.5 to match.Test plan
Verified end-to-end against a live Dispatcharr instance with real Zap2it/Gracenote credentials:
last_generatedstayed empty indefinitely,Auto-download tick… running download_epgfired roughly every 60 seconds continuously,/data/epgs/<file>.xmlnever updated past its initial creation date despite the underlying fetch itself succeeding repeatedly. After: exactly one tick fires per configured interval,last_generatedis correctly persisted, and the EPG source's programme data ingests successfully and advances (confirmed viaepg_programdatarow counts / max end_time).demo.tmsimg.comhas been running in production for about a week; thumbnails load correctly in Jellyfin's Live TV guide, no moreUnable to pre-cachewarnings.Diffed and reviewed manually (this repo ships the plugin as
zap2xml.zip, so a rawgit diffon the zip isn't reviewable in GitHub's UI — all behavioral changes are described in full above;channels.dband__init__.pyare untouched).🤖 Assisted by Claude Code