I write C++ and CUDA, mostly for LLM inference. I build systems from the ground up, check them against a trusted reference, and publish the numbers, including the ones that don't flatter me.
| CUDA backend of llama.cpp (100K+ ★) |
decode on a T4 from my own CUDA kernels |
max logit error vs HuggingFace |
p50 at 95.4% recall, my own HNSW index |
Note
Three systems built from scratch (an LLM inference engine, a vector database and a code analyzer) and two CUDA kernels merged into llama.cpp, one merged by the project's creator, Georgi Gerganov.
Contributing to the CUDA backend since Aug 2026.
| PR | What it does | How it was verified | Status |
|---|---|---|---|
| #27573 | New CUDA kernel for POOL_1D (average and max), closing a gap in GPU operator coverage |
Every kernel size, stride and padding combination: 216 cases on two T4s | ✅ Merged by Georgi Gerganov |
| #28897 | i16/i32 for GGML_OP_DUP on CUDA. Both were silently falling back to the CPU: fixed the capability check and added the missing i16 path |
Full backend suite: 16,097/16,097 passing, zero regressions | ✅ Approved by Georgi Gerganov, merged by am17an |
| # | Project | What it is | Headline result |
|---|---|---|---|
| 01 | verbum.cpp | LLM inference engine in C++20 & CUDA, from scratch | 28 tok/s on T4 · 3e-5 vs HF |
| 02 | Lattice | Embedded vector database with a hand-written HNSW index | 583 µs p50 @ 95.4% recall |
| 03 | RAAG | Parallel C++ code analyzer with AI refactoring guardrails | 3.69x on 8 cores · 1.1M AST nodes |
LLM inference engine, from scratch · C++20 CUDA Python
Runs Qwen3-0.6B end to end in C++20 and CUDA, with no PyTorch and no existing runtime. The safetensors loader, BPE tokenizer, grouped-query attention, RoPE, SwiGLU, KV cache and sampling are all hand-written.
flowchart LR
A[safetensors<br/>weights] --> B[BPE<br/>tokenizer]
B --> C[Embedding]
C --> D["28 x decoder layer<br/>RMSNorm · GQA attention · RoPE<br/>KV cache · SwiGLU"]
D --> E[LM head]
E --> F[Sampling]
F -->|next token| B
D -.->|CUDA kernels| G[(T4 GPU)]
| Result | |
|---|---|
| Correctness | Logits match HuggingFace to 3e-5. Diffing against that reference caught 5 silent bugs (RoPE convention, GQA mapping, KV-cache offsets) that gave wrong output without crashing |
| Speed | CUDA kernels (tiled matmul, RMSNorm, RoPE, SwiGLU, attention decode) decode at 28 tok/s on a T4, 38x over CPU on the same machine, ~25% of the memory-bandwidth roofline |
| Memory | Per-row INT8: 3.99x smaller weight matrices, all 196 tensors under 1.3% error, identical greedy output. Also found a duplicate lm_head wasting 622 MB |
Embedded vector database · C++20 HNSW Python
A vector database you link against, closer to SQLite than to a service. HNSW index, WAL storage engine, crash recovery and quantization, all written from scratch in C++20.
| Result | |
|---|---|
| Search | 583 µs p50 at 95.4% recall on SIFT10K, 1.3 ms p50 on SIFT1M. HNSW is 9.4x faster than brute force at 50K vectors |
| Storage | WAL with crash recovery, mmap segment files, atomic checkpoints. Concurrent path clean under ThreadSanitizer |
| Honesty | Benchmarked against Qdrant and Chroma, with the losses published too (build time is the big one) |
| Shipping | On PyPI as pylattice-db. 30 GoogleTest cases, CI that fails on benchmark regressions |
Architectural analytics platform · C++20 Python GraphRAG
Parses a codebase, builds its real dependency graph, and limits AI-assisted refactoring to the code a change can actually reach.
| Result | |
|---|---|
| Parsing | Parallel C++20 Tree-sitter parser on a std::jthread pool: 3.69x on 8 cores, 1.1M AST nodes from 579 files, zero failures |
| Analysis | Coupling, instability and cohesion across 913 dependency edges in nlohmann/json and fmt. Found a class in fmt with an LCOM4 of 55 |
| Guardrails | GraphRAG scoped to a change's blast radius. A GitHub Actions gate blocks risky PRs (above: a real one it blocked). 307 tests, 86% coverage |
| Shipping | VS Code extension that wraps the same CLI, so the editor and CI always agree |
| Languages | C++20 · CUDA · Python · SQL |
| GPU & performance | CUDA kernels · shared-memory tiling · roofline analysis · INT8 quantization · benchmarking · multithreading (std::jthread, std::atomic) |
| LLM inference | KV cache · grouped-query attention · RoPE · RMSNorm · SwiGLU · BPE tokenization · sampling · safetensors · llama.cpp/ggml · HuggingFace Transformers |
| Vector search & RAG | HNSW · scalar quantization · write-ahead logging · mmap storage · Qdrant · Chroma · RAG · GraphRAG |
| Backend & tools | CMake · Linux · Git · Docker · GitHub Actions · pybind11 · GoogleTest · ThreadSanitizer · Tree-sitter · FastAPI · SQLite · PyPI |
| CS fundamentals | Data structures & algorithms · OOP · SOLID · design patterns · low-level design |
Nine articles on Hashnode, also republished on Medium via Stackademic. Each one covers a real bug or a real measurement.
Show all articles
verbum.cpp
- What Actually Happens Inside a Transformer Forward Pass: five bugs that never crash and just give wrong answers
- INT8 Quantization the Second Time Around
- Two From-Scratch Systems, and the Day They Talked
Lattice
- Building an HNSW Index From Scratch
- Benchmarking Against Qdrant and Chroma
- What I Learned Building a Storage Engine From Scratch (and What I'd Change)
RAAG
Tip
- Diff against a reference. If there's a trusted implementation, I compare against it number for number before I believe my own output.
- Measure on the same machine. A speedup across two different computers isn't a speedup.
- Publish the honest number. If my system loses a benchmark, the loss goes in the README next to the win.
Currently: GPU performance work in open-source LLM inference engines.
Software Engineer (Independent) since Dec 2025 · B.Tech CSE, UPES (2024) · Before engineering, a year making music full-time.