Skip to content

fix: migrate to async SQLAlchemy and harden runtime robustness - #1

Open
mbuffa wants to merge 2 commits into
mainfrom
refac/async
Open

fix: migrate to async SQLAlchemy and harden runtime robustness#1
mbuffa wants to merge 2 commits into
mainfrom
refac/async

Conversation

@mbuffa

@mbuffa mbuffa commented Jul 1, 2026

Copy link
Copy Markdown
Owner
  • Replace sync SQLAlchemy engine/Session with async engine + AsyncSession (sqlite+aiosqlite) so DB calls no longer block the event loop in async routes and the worker
  • Add SQLite PRAGMA busy_timeout=30000 to prevent "database is locked" errors under concurrent writes; also set synchronous=NORMAL and foreign_keys=ON
  • Wrap blocking YouTubeTranscriptApi.fetch() in asyncio.to_thread() so transcript fetching no longer stalls every concurrent SSE stream
  • Fix SSE endpoint pinning a DB connection for the whole stream lifetime; now uses a short-lived session scoped to the slug lookup only
  • Add explicit db.rollback() before retry in worker exception path, preventing PendingRollbackError from masking the real failure
  • Replace swallowed exceptions with logger.exception() throughout worker and summarizer; sanitize client-facing error messages (no raw exception text leaked to browser)
  • Bound _progress/_listeners memory: prune empty listener lists on unsubscribe; schedule eviction of completed video state after a 60 s grace period via call_later
  • Simplify worker pool: remove misleading MAX_CONCURRENT+4/Semaphore pattern — spawn exactly MAX_CONCURRENT workers, one job per worker
  • Track worker tasks in a module-level set (prevents premature GC); add stop_workers() called from lifespan for clean shutdown on Ctrl-C
  • Update conftest and test_routes to use async fixtures and AsyncSession
  • Add aiosqlite>=0.20 and greenlet>=3.0 dependencies

- Replace sync SQLAlchemy engine/Session with async engine + AsyncSession
  (sqlite+aiosqlite) so DB calls no longer block the event loop in async
  routes and the worker
- Add SQLite PRAGMA busy_timeout=30000 to prevent "database is locked"
  errors under concurrent writes; also set synchronous=NORMAL and
  foreign_keys=ON
- Wrap blocking YouTubeTranscriptApi.fetch() in asyncio.to_thread() so
  transcript fetching no longer stalls every concurrent SSE stream
- Fix SSE endpoint pinning a DB connection for the whole stream lifetime;
  now uses a short-lived session scoped to the slug lookup only
- Add explicit db.rollback() before retry in worker exception path,
  preventing PendingRollbackError from masking the real failure
- Replace swallowed exceptions with logger.exception() throughout worker
  and summarizer; sanitize client-facing error messages (no raw exception
  text leaked to browser)
- Bound _progress/_listeners memory: prune empty listener lists on
  unsubscribe; schedule eviction of completed video state after a 60 s
  grace period via call_later
- Simplify worker pool: remove misleading MAX_CONCURRENT+4/Semaphore
  pattern — spawn exactly MAX_CONCURRENT workers, one job per worker
- Track worker tasks in a module-level set (prevents premature GC);
  add stop_workers() called from lifespan for clean shutdown on Ctrl-C
- Update conftest and test_routes to use async fixtures and AsyncSession
- Add aiosqlite>=0.20 and greenlet>=3.0 dependencies

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mbuffa mbuffa self-assigned this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant