Skip to content

fix: log every silently swallowed error, catch expected classes only (supersedes #14) - #17

Merged
Blackwellcode merged 13 commits into
masterfrom
audit/silent-failures-v2
Nov 12, 2026
Merged

Blackwellcode merged 13 commits into
masterfrom
audit/silent-failures-v2

Conversation

@Blackwellcode

@Blackwellcode Blackwellcode commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Supersedes #14 (rebased onto master after #15, plus the review fixes). The invariant tests from #14 moved to #16.

Summary

Broad exception handlers that hid failures without a log line now catch the expected class and log it. Fallback values are unchanged. One real bug is fixed.

 abort_deleted_media(streamer, filename)
   try:
-    if database.is_deleted_media(...): abort(404)   # NotFound is an Exception ...
-  except Exception: pass                           # ... so the 404 was swallowed
+    deleted = database.is_deleted_media(...)
+  except DB_ACCESS_ERRORS: rate-limited warning, serve as before
+  if deleted: abort(404)
core/database.py      DB_ACCESS_ERRORS = (sqlite3.Error, OSError, ValueError)   # get_db() makedirs, _normalize_channel_row int()
WEBUI/webui.py        _warn_rate_limited(site, ...)   one warning per minute and call site (+ repeat count)
                      _ChatLineSkips                  chat-log lines that are no JSON object: counted, reported once per read
core/storage/base.py  close_quietly(resource: Closable)  OSError / urllib3 HTTPError -> debug (reason in docstring)
core/redis_client.py  REDIS_ERRORS = (RedisError, OSError)
recorder/*            recording-loop checks stay broad on purpose, now with a throttled warning

Commits, in order:

  • storage close helper
  • token persistence and progress bookkeeping
  • core helpers
  • tombstone 404
  • webui lookups
  • changelog
  • test formatting
  • DB_ACCESS_ERRORS and rate limit
  • close_quietly move
  • core/recorder continue/return handlers
  • webui continue/return handlers
  • changelog

Evidence

  • Tombstone bug: DeletedMediaTombstoneTests.test_tombstoned_vod_page_is_404 returns 200 on master and 404 here.

  • Broad handlers without log or raise (AST count outside tests/ and docs/; throttled_log and _warn_rate_limited count as logging):

    pass continue constant return other total
    master a66f0ce 46 15 31 91 183
    this branch 77cc24a 0 0 0 89 89

    The 89 "other" handlers do something besides pass/continue/return (for example return Response("[]")). They are out of scope here.

  • Tests: python -m unittest discover -s tests -> master Ran 332 tests ... OK (skipped=20), branch Ran 377 tests ... OK (skipped=20) on Windows. The 20 skips are POSIX-only tests (process groups, real flock, file modes, symlinks); they run here only in CI.

  • Coverage, the same command on both trees:

    uv run --no-project --python <venv>\Scripts\python.exe --with coverage -- python -m coverage run --branch --source=. --omit=".venv/*,tests/*,docs/*" -m unittest discover -s tests
    uv run --no-project --python <venv>\Scripts\python.exe --with coverage -- python -m coverage report
    
    `<venv>` is the absolute path of the project virtualenv (the one the tests normally run in); both commands run from the repository root.
    master a66f0ce : TOTAL  21927  13700  7434  802  34%
    branch 77cc24a : TOTAL  22018  12843  7470  965  38%
    
  • Negative tests: an unexpected class (RuntimeError) still propagates. OSError and ValueError take the fallback and leave a log line. 5 failing media lookups produce 1 warning, and after 60 s a second one with "4 more". A list of 10 access checks produces 1 warning. twitch_watch with failing channel lookups returns 200 with 2 warnings. A failing progress save warns once per transfer and never aborts it.

  • Changed assertions in this PR's own tests (git diff 0e49208 HEAD -- tests/; no assertion from master is touched):

    • test_maintenance_bypass_check_failure_is_logged_and_denied is removed. It mocked is_user_exempt to raise, but that call cannot raise: site_maintenance.load_state() handles unreadable files itself, so the handler was dead and is gone. It is replaced by MaintenanceBypassTests, which uses a real corrupt maintenance.json.
    • assertLogs("storage.http") -> assertLogs("storage.base") because close_quietly moved modules.
    • WebuiLookupFailureTests.setUp clears the module-wide rate-limit state.
    • All other test diffs only add tests or re-wrap with lines.
  • ruff check .: All checks passed. compileall: OK.

Merge Danger

Door: two-way

Plain revert. No schema, config or data changes.

Blast Radius: moderate

  • An error class nobody expected (for example a TypeError from a code bug) now shows up as an HTTP 500 or a traceback, where it used to vanish. That is intended, but it can surface latent bugs right after deploy.
  • Database fallbacks catch sqlite3.Error, OSError and ValueError.
  • The recording loop's own checks stay broad, so they cannot stop a recording.
  • Deleted VODs and clips whose file still exists now return 404, as intended.
  • Log volume during a database outage is bounded to one line per minute and call site.

🤖 Generated with Claude Code

Blackwellcode and others added 12 commits November 12, 2026 16:35
Fifteen `try: response.close() except Exception: pass` blocks in the
storage HTTP layer and the Google Drive, Dropbox, OneDrive, WebDAV and S3
providers become http.close_quietly(). It catches only OSError (socket
already gone, requests' ConnectionError) and urllib3's HTTPError, logs
them at debug level with the provider name, and lets anything else
propagate. The transfer result is unchanged: closing only returns the
connection to the pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- base: a failing config-changed callback (rotated OAuth token not
  persisted) is caught as StorageError/sqlite3.Error/OSError/ValueError and
  logged as a warning; the request still succeeds as before.
- service: a locked database while saving transfer progress is logged at
  debug level; a failure to record the provider error after a failed
  restore is logged as a warning and the restore error is still raised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ss helpers

Ten `except Exception: pass` handlers outside storage and the web UI now
catch the specific failure and leave a log line; every fallback value is
unchanged:

- twitch_api/twitch_eventsub: an unreadable HTTP error body (OSError,
  http.client.HTTPException) is logged at debug level; the status code
  still decides the result. reload_env() calls redis_client.delete_state()
  directly because it already absorbs Redis errors.
- live_watch_diag: sqlite3.Error/ValueError reading channel configs is a
  warning; the diagnosis continues without the database fields.
- chat_chunker: an unreadable cached meta.json is a warning and the chunks
  are rebuilt.
- redis_client: failing to remove the dashboard shm cache is a warning.
- tube_auth: sqlite3.Error reading VOD access rows is a warning and the
  seed list is used, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abort_deleted_media() called abort(404) inside `try ... except
Exception: pass`. werkzeug's NotFound is an Exception, so the handler
swallowed it and a tombstoned VOD or clip whose file was still on disk
(restore, failed unlink, re-sync) kept opening in the player.

The tombstone lookup now catches only sqlite3.Error, logs it and serves
as before; abort(404) runs outside the handler. The alias lookup in the
same function logs sqlite3.Error instead of passing silently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The last sixteen pass-only broad handlers in webui.py (plus the adjacent
channel-config lookup in twitch_watch) catch the failure they guard
against and log a warning; every fallback stays as it was:

- channel/alias lookups (can_view_streamer, resolve_media_path,
  delete_video_file, list_all_streamer_folders, _recording_logins_set,
  twitch_watch, _vod_access_candidate_slugs): sqlite3.Error, plus OSError
  where the archive folder or recording state is read.
- Twitch GQL profile/user-id lookups: OSError (URLError, timeouts),
  ValueError (bad JSON) and http.client.HTTPException. Unexpected answer
  shapes are parsed by _twitch_gql_user() instead of raising
  AttributeError into the handler.
- maintenance bypass check: OSError/ValueError, user stays not exempt.
- title from the chat log in read_media_sidecar_metadata: OSError.
- can_view_streamer no longer wraps normalize_channel_name(), which is
  pure string work and cannot fail.

AST count of `except Exception/bare: pass` outside tests: 46 -> 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Thirteen context-manager continuations in the new error-handling tests
had lost their backslash-newline and ran on one 200+ character line.
Only line breaks change; no assertion or patch target differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…are rate-limited

Review of #14: handlers that caught only sqlite3.Error let OSError from
get_db() (it creates the data directory) and ValueError from
_normalize_channel_row() (non-numeric record flag) through as HTTP 500s.

- core.database.DB_ACCESS_ERRORS = (sqlite3.Error, OSError, ValueError);
  every database fallback in webui, tube_auth, live_watch_diag and the
  storage service catches exactly this tuple. Other classes propagate
  (negative test).
- webui._warn_db_fallback(): can_view_streamer, resolve_media_path and the
  tombstone check log at most once per minute and call site, with the
  number of repeats, instead of once per list entry or request.
- storage: a failed progress save warns once per transfer, then stays at
  debug; it never aborts the transfer.
- current_user_bypasses_maintenance(): the handler around is_user_exempt()
  was dead (site_maintenance.load_state() handles unreadable files and
  logs itself). It is removed. The mock-based test is replaced by a test
  with a real corrupt maintenance.json.
- Tests for twitch_watch with failing channel lookups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…justified

- close_quietly() moves from storage.http to storage.base with a Closable
  protocol type; s3.py no longer imports the HTTP helper just for it. Its
  log line now comes from the "storage.base" logger (tests follow).
- The remaining debug-level lines say in code why they are not warnings:
  close_quietly (routine connection resets, no outcome change),
  twitch_api._drain_http_error and the EventSub POST (the HTTP status is
  already logged at warning/info).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the web UI

Review of #14, point 3, for everything except webui.py:

- recorder: a failed salvage move (OSError) and an unreadable recording
  pointer (OSError/ValueError, throttled) are warnings. The recording-loop
  checks (is_process_running, live presence, EventSub, Helix, chat
  overlay) keep their broad handler on purpose, so a failure there can never
  stop a recording, but now log a throttled warning with the reason in code.
  The timestamp-format loop logs at debug, because trying the next format is
  its job.
- clip_engine/worker: ffprobe and frame extraction catch
  OSError/SubprocessError(/ValueError); recording pointers
  OSError/ValueError plus a dict check; a candidate file that vanishes
  between isfile() and getsize() is skipped explicitly; _is_direct_child
  fails closed with a warning.
- chat_chunker: unreadable lines are counted, not swallowed (warning for
  more than one, debug for a single half-written tail line); non-object
  lines no longer reach data.get().
- redis_client: REDIS_ERRORS = (RedisError, OSError); ping, delete_state
  and queue length log a warning.
- storage quotas (Dropbox, OneDrive, WebDAV): provider/parse errors are
  logged; the storage page still shows no quota.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t returns

Review of #14, point 3, for webui.py:

- _ChatLineSkips replaces ten `except Exception: continue` around chat-log
  JSON (media title, jump markers, chat index, global search, chat replay,
  live chat). Lines that are not a JSON object are counted and reported
  once per read: debug for a single half-written tail line, a rate-limited
  warning for more. Non-object lines no longer reach data.get(), where
  build_chat_jump_markers used to raise AttributeError.
- current_recording_info: OSError/ValueError, rate-limited warning.
- constant returns: rate-limit storage import (ImportError), user-rights
  check (RuntimeError outside a request at debug, DB_ACCESS_ERRORS as
  warning), log tail/remux/status/transcript files (OSError/ValueError),
  ffprobe (OSError/SubprocessError/ValueError; the 1 s per-file probe
  rate-limited), URL parsing (ValueError, fail closed, debug), sort keys
  and chat timestamps (TypeError/ValueError, debug), thumbnail URL
  (BuildError/DB_ACCESS_ERRORS; RuntimeError outside an app context at
  debug).
- _warn_db_fallback is now the general _warn_rate_limited.

AST count of broad handlers without log or raise, outside tests/docs:
origin/master 183 (pass 46, continue 15, constant return 31, other 91)
-> 89 (other 89).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Blackwellcode
Blackwellcode merged commit 9fbd488 into master Nov 12, 2026
4 checks passed
@Blackwellcode
Blackwellcode deleted the audit/silent-failures-v2 branch November 12, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant