Jaehee Kim · Pilsung Kang
📄 Paper · 🌐 Project Page · 🤗 Models · 🤗 Llama 3 Full Alien · 🧪 Recovery Evals · 💻 Code
AlienLM is a client-side privacy layer for black-box LLM APIs. It translates natural text into an Alien Language through a vocabulary-scale token bijection, adapts the target model with Alien Adaptation Training (AAT), and recovers the model output back into natural language on the client side.
This main branch is the lightweight entrypoint for AlienLM tokenizer
initialization and translation utilities. For the full ICML 2026 experiment
snapshot, including tokenizer assets, Axolotl configs, and evaluation launchers,
use the icml branch.
- 📄 Paper: arXiv:2601.22710
- 🌐 Project page: kimjaehee0725.github.io/AlienLM
- 🤗 Model collection: dsba-lab/AlienLM
- 🤗 Llama 3 Full Alien checkpoint: dsba-lab/llama3-8b-instruct-alienlm-full
- 🧪 Recovery evaluations: KimJaehee0725/AlienLM-recovery-evals
- 💻 Official code: KimJaehee0725/AlienLM
| Branch | Purpose |
|---|---|
main |
Minimal, stable entrypoint for tokenizer initialization and translation |
icml |
ICML 2026 paper artifact snapshot with assets, training configs, and evaluation launchers |
The main branch intentionally excludes checkpoints, raw evaluation dumps,
dataset caches, W&B runs, and large experiment-only files. Recovery and
robustness experiments are maintained separately in
AlienLM-recovery-evals.
| Component | What it is for | Start here |
|---|---|---|
| Reversible token codec | Length-preserving token-ID permutation for direct model input | alienlm/codec/ |
| Lossless JSON transport | Checksummed text transport for AlienTokenSequence |
alienlm/transport/ |
| Translator | Lossless wire translation plus explicit legacy text utilities | translator/translator.py |
| Tokenizer initialization | Token matching and randomized token reordering utilities | tokenizer/token_init/ |
| Smoke checks | Small local checks that do not require paper-scale compute | scripts/smoke/ |
| Paper artifacts | Tokenizer assets, AAT configs, and evaluation launchers | icml branch |
The future-facing package layout and the shared vocab/train/evaluation artifact
contract are documented in docs/architecture.md.
The recommended setup uses uv. Run all commands
from the repository root.
git clone https://github.com/KimJaehee0725/AlienLM.git
cd AlienLM
uv syncInstall optional dependencies when constructing alien token orderings:
uv sync --extra init --extra freqFor gated Llama-family tokenizers or checkpoints, log in to Hugging Face first:
uvx --from huggingface_hub huggingface-cli loginTokenizerTranslator now uses a checksummed JSON token payload by default:
plain2alien: plain text -> alien token IDs -> JSON payloadalien2plain: JSON payload -> inverse token mapping -> plain text
When no permutation artifact is supplied, the translator derives the one-to-one mapping from the base and alien tokenizer vocabularies. The two tokenizers must contain the same token strings with different ID assignments, and special token IDs must remain fixed.
Download the tokenizer files for the Llama 3 Full Alien checkpoint:
mkdir -p assets/llama3-8b-instruct-alienlm-full
uvx --from huggingface_hub huggingface-cli download \
dsba-lab/llama3-8b-instruct-alienlm-full \
tokenizer.json tokenizer_config.json special_tokens_map.json \
--local-dir assets/llama3-8b-instruct-alienlm-fullThen produce a wire payload and restore it after storage or an exact echo:
from pathlib import Path
from translator import TokenizerTranslator
translator = TokenizerTranslator(
alien_tokenizer_path=str(
Path("assets/llama3-8b-instruct-alienlm-full").resolve()
),
opensource_tokenizer="meta-llama/Meta-Llama-3-8B-Instruct",
)
plain = "All happy families are alike; each unhappy family is unhappy in its own way."
payload = translator.plain2alien(plain)
restored = translator.alien2plain(payload)
print("plain:", plain)
print("payload:", payload)
print("restored:", restored)
assert restored == plainThis transport path is lossless because it never decodes alien IDs into a flat
string. It is intended for a server or storage layer that returns the payload
unchanged. Sending this JSON to a text-only LLM does not inject its token_ids
as model input IDs; the provider will tokenize the JSON as ordinary text.
The paper-compatible text path remains available under explicit legacy names:
alien_text = translator.plain2alien_text_legacy(plain)
restored = translator.alien2plain_text_legacy(alien_text)This legacy path can lose token boundaries because tokenizer decoding is not injective. Do not use it when exact round-trip recovery is required.
When local inference or evaluation exposes generated IDs, do not convert them through the original tokenizer first. Decode them directly:
plain_response = translator.decode_token_ids(
generated_ids,
skip_special_tokens=True,
)A legacy black-box path may instead apply its fixed original tokenizer and return only decoded text. For that case, AlienLM retains the experimental recovery path:
plain_response = translator.recover_server_response(server_text)This method builds the inverse decoder lattice, keeps minimum-token candidate segmentations, and ranks them using tokenizer-internal signals only. It does not require token IDs, logprobs, changes to the server tokenizer, or another language model. The current implementation supports the ByteLevel decoder used by Llama 3 and the Replace/ByteFallback/Fuse decoder used by Gemma 2.
Recovery remains heuristic because distinct ID sequences may decode to exactly the same server text. It is intended for short responses; long-form output, invalid byte sequences rendered as the replacement character, and multiple equally plausible candidates may not be recovered exactly. Applications that require strict reversibility must obtain token-level metadata or use a reversible transport protocol.
For the Llama 3 Full Alien example above, the legacy decoded-text output follows the model card:
Natural text
All happy families are alike; each unhappy family is unhappy in its own way.
Alien text
One unhappyamilies
hike..:
happy
happy hodin waypoints,
Original token IDs
[2460, 6380, 8689, 527, 27083, 26, 1855, 43251, 3070, 374, 43251, 304, 1202, 1866, 1648, 13]
Alien token IDs
[4054, 43251, 60004, 66417, 35331, 114100, 27381, 6380, 39185, 23136, 6380, 109132, 8299, 21649, 82386, 11]
The lossless JSON translation can be run from the command line:
ALIEN_TOKENIZER="$(pwd)/assets/llama3-8b-instruct-alienlm-full"
uv run python translator/translator.py \
--alien-tokenizer-path "$ALIEN_TOKENIZER" \
--opensource-tokenizer meta-llama/Meta-Llama-3-8B-Instruct \
--direction plain2alien \
"All happy families are alike; each unhappy family is unhappy in its own way."Use --direction plain2alien-text-legacy for the old visible Alien text and
--direction recover-server-response for its heuristic response recovery.
Run the dependency-light round-trip smoke test:
uv run python scripts/smoke/translator_roundtrip.pyAlienLM builds an alien tokenizer by matching and reordering token IDs while keeping the base tokenizer's ID space compatible.
uv run python tokenizer/token_init/token_matching.py \
--base_model meta-llama/Meta-Llama-3-8B-Instruct \
--proxy_model Qwen/Qwen2.5-7B-Instruct \
--token_freq_json /path/to/token_freq.json \
--output matches-sim-and-diff.txtFor randomized bucket reordering, use:
uv run python tokenizer/token_init/token_random.py --helpThe full paper snapshot on the icml branch includes additional tokenizer
assets and experiment launchers.
tokenizer/token_init/: alien language initialization utilitiesalienlm/codec/: reversible token-ID representation and mapping artifactsalienlm/transport/: checksummed JSON wire formattranslator/: lossless wire translation and explicit legacy text translationscripts/smoke/: lightweight local smoke checks
If AlienLM helps your research, please consider citing:
@inproceedings{kim2026alienlm,
title={AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs},
author={Kim, Jaehee and Kang, Pilsung},
booktitle={Proceedings of the 43rd International Conference on Machine Learning},
year={2026}
}