Repository navigation
Samsung Exynos AI LiteCore - Introduce unified LLM runner for LLM models - #23799
Open
Jiseong-oh wants to merge 17 commits into
Open
Jiseong-oh wants to merge 17 commits into
Jiseong-oh wants to merge 17 commits into
Conversation
Jiseong-oh
requested review from
kirklandsign and
larryliu0820
as code owners
October 14, 2026 04:19
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23799
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit a22d8e2 with merge base 18e0cc0 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
|
|
Jiseong-oh
added
partner: samsung
This PR needs a
|
Collaborator
Author
|
/easycla |
Jiseong-oh
force-pushed
the
unified-llm-runner
branch
from
October 14, 2026 10:03
42dc209 to
98146ca
Compare
Jiseong-oh
force-pushed
the
unified-llm-runner
branch
from
October 15, 2026 09:58
98146ca to
f623f7d
Compare
Jiseong-oh
force-pushed
the
unified-llm-runner
branch
from
November 2, 2026 11:27
f623f7d to
6bf0a5d
Compare
Add a prototype runner that executes an ENN backend Llama program with its KV cache managed on the host. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com>
Load the program together with its .ptd weights so the runner can use programs whose weights are stored externally. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: brucekk.kim <brucekk.kim@samsung.com>
Keep the weight buffer alive for as long as the debug logger reads it. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com>
Methods that share weights reuse one buffer through a cache keyed by the weight-buffer hash id, and their models are opened from the file descriptor with the external weights. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: Hoon Choi <hoon98.choi@samsung.com>
Add the runner flow and the IO manager that prepares the token, mask, RoPE and KV-cache inputs and folds the KV outputs back into the cache. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com>
Rework how the IO manager fills the attention mask buffer. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: MJ Kim <mj.kim010@samsung.com>
decoder.h derives from TextDecoderRunner, so include its header directly instead of relying on a transitive include. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com>
Fix lint errors across the runner, then add Gemma support: a positional embedding class, attention mask and position id handling for sequences longer than 128 tokens, outputs aligned with the PyTorch reference, and sample run scripts for Gemma 3 and Llama 3. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: MJ Kim <mj.kim010@samsung.com>
Build the runner around a config parser and restructure Decoder and LlmIOManager around it. This also fixes a heap corruption in Decoder::register_method and lifetime issues in LlmIOManager, and renames the LLM executor source. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: brucekk.kim <brucekk.kim@samsung.com>
EnnLogImpl no longer crashes now that it uses __android_log_vprint, a failed rotary embedding load returns an explicit error, and build and prompt template bugs are fixed. The Samsung llama runner is renamed to llm runner. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: Hoon Choi <hoon98.choi@samsung.com>
RunnerConfig now carries the model arguments, including those parsed from the Gemma 4 JSON. RoPE tables can be loaded from a binary file with precomputation as the fallback, the decoder resolves the position, mask and KV tensor indices and ranks, and the llama3 config and the prefill input resizing are fixed. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: brucekk.kim <brucekk.kim@samsung.com>
Fix the Gemma 4 text decoder I/O, the partial-RoPE table math and the chat template separator, precompute the RoPE position embeddings, and fix a double start_pos increment in multi-chunk prefill. A vision encoder adds multimodal prefill. The run script gains --quiet, the runner reprints the full answer at the end behind a verbose toggle, and the runner is exported as a library other sample apps can link. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: MJ Kim <mj.kim010@samsung.com>
Let callers stop an in-flight generation. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: brucekk.kim <brucekk.kim@samsung.com>
Add a sample CLI that consumes the public C API, and rename the ennllm library to exynos_llm_wrapper. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: MJ Kim <mj.kim010@samsung.com>
The ENN perf mode is set from the detected SoC. enn_executor_llm_runner drives the runner through the exynos_llm_wrapper C API and gains a system prompt, --prompt_file, --system_prompt_file, --max_token_limit, configurable prompt echo and a thread-safe stop. The Gemma 4 prefill sliding-window mask is fixed, BOS is prepended to Gemma 4 text prompts, and the Samsung Android build gets a --vulkan option. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: Hoon Choi <hoon98.choi@samsung.com>
Let callers feed a preformatted prompt verbatim or prefill the start of the reply. Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com> Co-authored-by: MJ Kim <mj.kim010@samsung.com>
- Fix memory corruption and implement refcounting in enn_backend.cpp - Resolve buffer overflow issues in IO manager and RoPE embeddings - Fix Tensor lifetime and model loading logic in decoder.cpp - Improve config validation and optimize build system for LLM runner - General cleanup: remove debug logs and fix CLI bugs Signed-off-by: Jiseong Oh <jiseong.oh@samsung.com>
Jiseong-oh
force-pushed
the
unified-llm-runner
branch
from
November 3, 2026 11:23
6bf0a5d to
a22d8e2
Compare
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unified model-specific LLM runners into a single runner using a config-driven approach. This enables support for multiple LLM models (currently Llama and Gemma 3/4) within a single runner and provides an extensible framework for adding further model support in the future.
Test plan
./backends/samsung/build.sh --build android
./examples/samsung/build.sh --llm-wrapper-so
push libexynos_llm_wrapper.so /data/local/tmp
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/data/local/tmp
enn_executor_llm_runner
--model_path=${LLM_MODEL_PATH}
--data_path=${LLM_DATA_PATH}
--tokenizer_path=${LLM_TOKENIZER_PATH}
--pos_embedding_path=${LLM_POS_EMBEDDING_PATH}
--config_path=${LLM_CONFIG_PATH}
--seq_len=${LLM_SEQ_LEN}
--temperature=${LLM_TEMPERATURE}
--num_bos=1
--max_token_limit=${LLM_MAX_TOKEN_LIMIT}
--prompt="What is the capital of Korea?"
cc @SS-JIA @digantdesai @kimishpatel