TokenSandbox

Train a BPE tokenizer on your own domain-specific text and measure exactly how much more efficient it is than general-purpose tokenizers (GPT-4, Llama) for your data — in real token-count and estimated cost savings.

Status

🚧 Early / actively being built in public. Not yet ready for real use.

Roadmap

Phase 1 — Core BPE engine (in progress)

• get_word_frequency — corpus → word frequency counts
• get_pair_counts — count adjacent symbol pairs, weighted by frequency
• merge_word — apply a single merge rule to one word
• train_bpe — full training loop, produces vocab + ordered merge list
• BPETokenizer class — wraps vocab + merges with .encode(), .save(), .load()
• Basic regex-based pre-tokenization (beyond naive whitespace split)
• Unit tests against hand-verified toy corpus examples

Phase 2 — Comparison against production tokenizers

• compare_tokenizers(eval_text, tokenizers: dict) — accepts any object exposing .encode(text) -> list, so custom/third-party tokenizers can be compared without touching TokenSandbox's internals (duck-typed, not tied to BPE specifically)
• Wrapper around tiktoken (GPT-4/GPT-3.5) exposing the same .encode() shape
• Wrapper around Hugging Face tokenizers (BERT WordPiece, Llama SentencePiece)
• Compression ratio + estimated cost/context-window savings report
• Held-out evaluation set support (train/eval split, not scoring on training data)

Phase 3 — Usability

• CLI: tokensandbox train --corpus my_docs.txt --vocab-size 8000
• CLI: tokensandbox compare eval_docs.txt --tokenizer my_tokenizer/ --against gpt-4
• Example corpora bundled in repo so people can try it without their own data
• Quickstart in README with real, runnable example + sample output

Phase 4 — Stretch goals

• Byte-level base alphabet (currently character-level; won't gracefully handle non-ASCII input until this lands — documented limitation for now)
• Interactive step-by-step merge visualizer (teaching aid)
• Support training additional algorithms (WordPiece, Unigram) via a shared training interface — deferred until a second concrete algorithm justifies the abstraction; comparison already supports arbitrary tokenizers today

Design notes

• Comparison is intentionally decoupled from training: any tokenizer — BPE, a Hugging Face model, or something entirely custom someone writes themselves — can be compared as long as it exposes .encode(text) -> list[str]. No shared base class or interface required for this.
• A smaller token count on your domain doesn't mean "objectively better tokenizer" — it means better fit for that domain. A custom tokenizer will typically lose on general text outside what it was trained on.

Non-goals (for now)

• Not trying to be a faster or production-grade BPE implementation — tiktoken and Hugging Face tokenizers already do that well.
• Not trying to be a general-purpose multi-tokenizer visualizer — several good tools already exist for that.

Prior art / inspiration

• tiktoken — production BPE, and its _educational submodule
• minbpe — educational BPE implementation
• Research on tokenizer domain adaptation and compression ratio (NSL metric)