fix(models): race-free lazy encoder load; timeout and length check for model downloads

[kevincostner17]

Summary

• OnnxEncoder lazy load race: concurrent encode_texts hits a None tokenizer #340 – OnnxEncoder lazy-load race. _load checked only _session outside the lock and set _session before loading the tokenizer, so a concurrent encode_texts could call encode_batch on None. The session and tokenizer are now built as local variables and assigned tokenizer first, session last. The check outside and inside the lock requires both. A failed tokenizer load leaves the encoder unloaded, so the next call retries.
• fd.models.pull installs truncated downloads and has no network timeout #341 – fd.models.pull truncation and timeout. _fetch now passes timeout= to urlopen (from FRESHDATA_MODEL_TIMEOUT, default 60 s; a non-numeric, non-finite or non-positive value raises ValueError). It counts the bytes written and raises OSError("incomplete download of <url>: got N of M bytes") when the count differs from Content-Length. The existing cleanup in pull() removes the .part file, so nothing is installed. Checksum handling and the URLs fetched are unchanged.
• Documented FRESHDATA_MODEL_TIMEOUT in docs/semantic-models.md.

Note: freshdata models pull on the CLI still only catches ModelError, so a truncated download or invalid timeout ends in a traceback there, as network errors already did.

Tests

New tests/test_models_download_robustness.py, fully offline:

• Fake onnxruntime/tokenizers modules injected through sys.modules. A second thread is started while the first is still inside Tokenizer.from_file; it waits instead of crashing, and the model loads once. There is also an 8-thread Barrier test, a check that a session without a tokenizer triggers a load, and a retry after a failed load.
• A fake urllib.request.urlopen response covering: a truncated body raises OSError and leaves no final file, no .part file and installed=False; a full download over several read chunks installs; a response without Content-Length is accepted; the default and overridden timeouts are passed to urlopen; invalid timeout values raise ValueError before any network call; a timeout error leaves no .part file.
• 13 of the new tests fail on the old code.

Verification

• ruff check . clean; mypy src/freshdata clean (202 files)
• Python 3.12: 4262 passed, 6 skipped
• Python 3.9: 4258 passed, 10 skipped

Closes #340
Closes #341

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 577426c9-50f6-4a48-9b84-bd13958813f4

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)