MoE streaming: expert reads in 256 KiB chunks by 16 threads (pin llama.cpp-vulkan cecf3ee)

[bong-water-water-bong]

• Pins third_party/llama.cpp-vulkan to cecf3ee (fork PR Docs: the 1bit engine runs inside Lemonade #23). Expert parts are now read in 256 KiB chunks by 16 threads, the streamer's new default (ONEBIT_MOE_CHUNK_KB, ONEBIT_MOE_IO). ONEBIT_MOE_STATS reports pread and copy time per read.
• moe/ExpertCache gains read_chunk and the read timing, matching the fork's copy. 1bit moe-cache --chunk-kb N is added.
• docs/moe-streaming.md: on Flash-Next, the gate-ahead prefetch doubles decode time (the drive is already the limit), and the reads-in-flight measurements are recorded.

Flash-Next UD-Q4_K_XL at 4608 slots: 143-151 s per 512-token pass, against 174-177 with whole-part reads (two interleaved rounds, perplexity unchanged). On Coder-30B the difference is within the noise.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit 70000b3

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 70000b3)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 23 - Partially compliant Compliant requirements: README intro and status updated docs/lemonade.md updated docs/serve.md updated PORTING step 1 and diagram updated docs/npu.md updated serve/main code comments updated Non-compliant requirements: Thank-you section not updated (not in diff) Keep 1bit serve usage as secondary note (not in diff) Requires further human verification: Verify that the thank-you section was updated in the README Verify that the secondary note about 1bit serve usage was added

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Chunked Read Logic The new chunked read logic in ExpertCache::start_load introduces a potential performance regression if opt_.read_chunk is set to a value that results in very small chunks. This could lead to a high number of read operations and increased overhead from managing many small reads, especially when the chunk size is not a multiple of the page size or when the overlap calculation results in many small read requests. const uint64_t chunk = opt_.read_chunk ? up(opt_.read_chunk) : 0; for (size_t k = 0; k < parts.size(); ++k) { const auto& p = parts[k]; const uint64_t off = p.base + uint64_t(e) * p.stride; const uint64_t a = down(off), b = up(off + p.bytes); uint8_t* dst = lay.base + lay.part_off[k] + size_t(s.index) * lay.part_bytes[k]; // [a, b) aligned for O_DIRECT, in chunks; each copies its overlap with [off, off + bytes) const uint64_t step = chunk ? chunk : b - a; for (uint64_t c0 = a; c0 < b; c0 += step) { const uint64_t c1 = std::min(b, c0 + step); const uint64_t lo = std::max(c0, off), hi = std::min(c1, off + p.bytes); if (lo >= hi) continue; Read r{si, fds_[p.file], c0, c1 - c0, dst + (lo - off), lo - c0, hi - lo}; (demand ? demand_q_ : prefetch_q_).push_back(r); s.pending++; } Timing Measurements The timing measurements for pread and memcpy operations in the reader() function are recorded per read operation, which could lead to a significant overhead in the timing code if many small reads are performed. This overhead could impact performance measurements and potentially affect the actual performance of the system. const auto t0 = std::chrono::steady_clock::now(); uint64_t done = 0; while (done < r.len) { const ssize_t n = ::pread(r.fd, bounce + done, r.len - done, (off_t) (r.off + done)); if (n <= 0) { if (n < 0 && errno == EINTR) continue; // short read at the end of the file: the window's tail past EOF is padding std::memset(bounce + done, 0, r.len - done); break; } done += (uint64_t) n; } const auto t1 = std::chrono::steady_clock::now(); std::memcpy(r.dst, bounce + r.lead, r.bytes); const auto t2 = std::chrono::steady_clock::now(); { std::lock_guard<std::mutex> lk(mu_); st_.read_ms += std::chrono::duration<double, std::milli>(t1 - t0).count(); st_.copy_ms += std::chrono::duration<double, std::milli>(t2 - t1).count(); st_.reads++; Slot& s = slots_[r.slot]; if (--s.pending == 0) s.state = State::Ready; }

[bong-water-water-bong]

CI's new registry_pins test (#165) failed here: the pin moved to cecf3ee while registry/architectures.json still recorded 546800c. I pushed 70000b3, which regenerates the registry at the committed pins. Counts are unchanged (323 mapped), since the fork change only touches the MoE-streaming files. For hand-made pin bumps from now on: scripts/registry-regen.sh does the same thing.

[github-actions]

Preparing review...

[github-actions]

Persistent review updated to latest commit 70000b3