Skip to content

Fix jsonl read bottleneck - #164

Merged
JojiiOfficial merged 2 commits into
devfrom
fix-client-bottleneck-from-file
Aug 13, 2026
Merged

Fix jsonl read bottleneck#164
JojiiOfficial merged 2 commits into
devfrom
fix-client-bottleneck-from-file

Conversation

@JojiiOfficial

@JojiiOfficial JojiiOfficial commented Aug 13, 2026

Copy link
Copy Markdown
Member

Load dataset's queries at startup into memory, so they don't bottleneck during search.
I ran into this during my experiments despite using a fast nvme.

The Issue

Jsonl reader costs a full File::open plus a full serde_json parse for each query. On H&M dataset for example, queries are quite large ~45kb.

Benches

bfb with -p 12 -t 12

client time per request concurrent achieved (of 12, median) searches/s
dev 1.005 ms 7.0 5,478
PR 0.291 ms 10.7 15,092

Difference in real world benchmark (two runs, same server and search setup).

binary median qps avg server time concurrency achieved* recall@10
dev 4,757 0.63 ms ~3.0 of 32 0.92079
PR 7,250 4.12 ms ~29.9 of 32 0.92079

* qps x latency

Costs

Loading it at startup is expensive and can take a few seconds depending on disk.
For reference, H&M query set takes 850ms on my machine, but my storage is on the faster side.
I still think it's better to unconditionally (= not based on any configurations) fix this bottleneck so we can rely on measurements being correct.

@JojiiOfficial
JojiiOfficial force-pushed the fix-client-bottleneck-from-file branch 2 times, most recently from 3004399 to 82f56fe Compare August 13, 2026 15:43
@JojiiOfficial
JojiiOfficial merged commit 8b6dec0 into dev Aug 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants