fix(http): make the sqlite request cache thread-safe

[Baz (bazarnov)]

Overview

👉 TL;DR: Stops the HTTP response cache from failing or crashing a sync when several worker threads use it at once. Every read and write of the cache now goes through one lock, so only one thread uses the cache database at a time. Specifically, monkey_patched_get_item now holds the backend lock while it reads, and HttpClient._request_session gives every requests_cache SQLiteCache it builds one process-wide _SQLITE_CACHE_LOCK.

Changes

• monkey_patched_get_item, the CDK's existing patch of requests_cache.SQLiteDict.__getitem__, runs its SELECT under self._lock and deserializes after releasing it. The docstring now explains the race and when the patch can go. The inline-key query from fix: monkey patch SQLiteDict in order to fix sqlite3.InterfaceError #725 is unchanged.
• New module-level _SQLITE_CACHE_LOCK (threading.RLock). HttpClient._request_session builds each SQLiteCache under it, since the constructor creates the tables, and assigns it to the backend's responses and redirects. One shared lock also covers runs without REQUEST_CACHE_PATH, where every cache uses one in-memory database and a connection touching a table another one is writing fails at once with database table is locked. A ponytail: comment names the ceiling: one lock per process, one lock per sqlite path if contention ever shows.
• Two tests in test_http_client.py:
  • test_cache_read_holds_the_lock_shared_by_every_cache (deterministic): two clients share one lock, and a read holds it through execute, fetchone and close.
  • test_concurrent_cached_sends_do_not_fail[file|in_memory]: 2 clients with the same name, 8 threads, 50 requests each, mixing cache hits and writes.

Review Spotlight

Reviewers with limited time, please review first:

• http_client.py#L84-L101: reads hold the lock, deserialization stays outside it
• http_client.py#L106-L110: why every backend shares one lock, and its ceiling
• http_client.py#L225-L228: the backend is built under the lock and wired to it
• test_http_client.py#L769-L801: deterministic test; each half of the fix alone still fails it

Root cause

The race is in requests_cache: each table has one sqlite3 connection shared by every thread (check_same_thread=False), and only writes take the lock, so reads run on a connection another thread is using (requests-cache#1190, still open). Several SQLiteCache instances on one file, which the issue also blames, are not a cause on their own: one client used by 8 threads fails by itself, while 2 clients with one thread each never fail.

Production syncs surfaced it as EOFError: Ran out of input, sqlite3.DatabaseError: no more rows available and sqlite3.OperationalError: database table is locked. SQLite builds without serialized mode, such as the macOS system library, crash the interpreter instead.

Risks

• Cache access is serialized per process. All sqlite calls made by CDK request caches in one process now run one at a time. The lock is never held across network I/O or (de)serialization. On a 4 clients x 8 threads x 100 requests x 10 runs workload (macOS, 3 rounds), the shared lock took 8.5-8.9 s against 9.1-9.6 s with per-backend locks plus the read lock.
• Who sees it: 244 connectors on airbyte master go through a cached HttpClient. 78 of them are directly exposed to the race (a cached parent that is also a top-level stream with more than one partition), for example source-github, source-gitlab, source-jira and source-zendesk-support. Counted by resolving all 535 connector manifest.yaml files with the CDK ManifestReferenceResolver (240 connectors with an auto-cached SubstreamPartitionRouter parent), plus git grep for explicit use_cache: true (9), PropertiesFromEndpoint (1) and Python use_cache (4).
• Output is unchanged. Records, state, schemas and messages stay the same; only the intermittent cache errors and crashes stop.
• Ceiling with requests-cache 1.3.x (58 of 163 connector poetry.lock files on master; the CDK locks 1.2.1): a write that fails with anything other than OperationalError leaves its transaction open, and later writes on that connection retry forever. That already hangs the sync today; with one lock it blocks every cache in the process rather than the caches on one file. The race fixed here was the realistic source of such errors.

Test plan

• poetry run pytest unit_tests/sources/streams/http/test_http_client.py -k "lock_shared_by_every_cache or concurrent_cached_sends" on macOS (Python 3.11, system SQLite 3.51 without serialized mode):

  • fix: 3 passed. The stress test passed 10 runs out of 10, about 0.5 s per case.
  • main source (git checkout origin/main -- airbyte_cdk/sources/streams/http/http_client.py): the deterministic test fails on the shared-lock assertion. The stress test failed in all 28 runs (14 per case): it crashes the interpreter (SIGSEGV, SIGILL, SIGBUS, SIGABRT), hangs, or fails with errors such as DatabaseError('file is not a database').
  • Each half of the fix alone fails the deterministic test: read lock only fails first.responses._lock is second.responses._lock, shared lock only fails assert [False, False, False] == [True, True, True]. Moving fetchone() and close() out of the locked block fails it with [True, False, False].

• The same three tests in the airbyte/source-declarative-manifest:7.30.0 image (Python 3.13.14, SQLite 3.40.1 serialized, requests-cache 1.2.1), 10 runs each:

  test	main	fix
  test_cache_read_holds_the_lock_shared_by_every_cache	10/10 failed	0/10 failed
  test_concurrent_cached_sends_do_not_fail[file]	6/10 failed: EOFError('Ran out of input'), DatabaseError('no more rows available')	0/10 failed
  test_concurrent_cached_sends_do_not_fail[in_memory]	10/10 failed: OperationalError('database table is locked')	0/10 failed

• poetry run pytest unit_tests/sources/declarative/test_concurrent_declarative_source.py -k test_read_with_concurrent_and_synchronous_streams, 5 runs on macOS: main crashed the interpreter in 4 and failed in 1; the fix passed all 5.

• Ad-hoc stress script (cached clients with the same name, 8 threads, 60 mixed requests each, 10 runs): 0 failed runs with the fix, file and in-memory each: 4 clients on macOS, and 2 and 4 clients in the Linux image with requests-cache 1.2.1 and 1.3.3.

• poetry run pytest unit_tests/sources/streams/http unit_tests/sources/streams/test_call_rate.py unit_tests/test_requests_cache_monkeypatch_version.py: 556 passed. poetry run pytest unit_tests/sources/declarative/test_concurrent_declarative_source.py: 89 passed.

• ruff format, ruff check and mypy are clean on the changed files.

Fixes #1150

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@bazarnov/fix-http-cache-sqlite-thread-safety#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch bazarnov/fix-http-cache-sqlite-thread-safety

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 985 tests  +3   4 973 ✅ +3   11m 13s ⏱️ + 1m 40s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 2e05226. ± Comparison against base commit d214b0c.

[github-actions]

PyTest Results (Full)

4 988 tests  +3   4 976 ✅ +3   12m 43s ⏱️ - 3m 21s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 2e05226. ± Comparison against base commit d214b0c.