Add SigLIP 2 zero-shot image classification and the Sort Photos demo - #18
Merged
Merged
Conversation
SigLIP2Manager loads the mobius-converted image and text encoders, embeds labels once, and scores images with Core ML's async API. SigLIP2Tokenizer ports Gemma BPE (identical ids to the Python tokenizer on 1,018 prompts incl. accents, CJK, emoji); the preprocessor reproduces PIL's antialiased bilinear resize. ImageSortCheck: zero-shot Oxford-IIIT Pets, 37 breed prompts, all 3,669 test photos: 94.85% (Python Core ML 94.77%), 195 photos/s with 4 in flight on an M5 Pro. ImageSortDemo animates photos into breed buckets (Show / Turbo, IMAGE_SORT_AUTOPLAY); Turbo sorts 1,000 photos in 5.2 s. Models load from SIGLIP2_MODEL_DIR until the Hugging Face repo exists.
Prints each decision with the label's softmax share among the 37 breeds and the model call time, matching SortAnythingDemo's SORT_LOG.
Each sorted photo becomes a tile in its breed's row (one bitmap, so thousands of tiles stay cheap); wrong calls get a red frame. The left panel shows the latest photo with its five most likely breeds. Header and board adapt to narrow windows. A cache holding the train split samples up to 7,349 photos (Turbo: 38 s, 192 photos/s, 94.3% on an M5 Pro). IMAGE_SORT_WAIT=1 waits for Start.
Name the animal (default): 26 Wikimedia Commons animal clips, every frame scored against the 26 names; caption, top-5 panel, and a Spotted strip; correct-frame rate from per-clip boundaries. M5 Pro: 31 fps, 9.5 ms per frame, 94.2% of frames, 26/26 species. Potato scene: 6x4 grid over a USDA sorting video, about 8 grids/s. GridClassifier gains top-5 and a prompt template.
Show mode flies every photo and lands its tile on arrival; Turbo flies two photos per 50 ms flush while the chart updates at full speed (7,349 photos in 39 s, 187 photos/s, 94.3%).
… narrow The empty chart picks how many tiles to stack per breed row (1-10) so the bitmap fills the available space; windows under 980 pt put the current photo and its top five above the chart.
…l log Removes Show mode, the mode picker, and Pace. Stats show photos per second and wall-clock ms per photo; the caption explains the four photos processed in parallel. The terminal log prints one photo per update with a +N more count, so printing does not slow the run under screen recording.
SigLIP2Manager.imageEncoderMilliseconds times the image model alone (median of 30 sequential calls after 5 warmups); the demo measures it at launch (5.2 ms on an M5 Pro) and shows it next to the live photos/s and ms per photo.
Neural Engine ms is measured live from each Core ML call's start and end: end minus the later of its start and the previous call's end, which removes queueing when calls overlap (median of the last 200). Stats are equal-width centered tiles that start at zero; buttons sit beside the title in narrow windows.
SigLIP2ModelStore fetches FluidInference/siglip2-base-patch16-256-coreml at a pinned revision with SHA-256 checks; SIGLIP2_MODEL_DIR still loads a local conversion. ImageSortCheck scores the 3,669 test photos by default (--split=all for 7,349): 94.85% / 94.26% at 202 photos/s from the downloaded packages. VideoSortDemo moves to its own branch (its videos are local only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds on-device zero-shot image classification with SigLIP 2 (base, 256 px) on Core ML, plus a photo-sorting demo.
SigLIP2Manager/SigLIP2ModelStore: loads the image and text encoders from FluidInference/siglip2-base-patch16-256-coreml (pinned revision, SHA-256 checked);SIGLIP2_MODEL_DIRloads a local conversion.SigLIP2Tokenizer: Swift Gemma BPE, token ids identical to the Python tokenizer on 1,018 prompts (accents, CJK, emoji).SigLIP2ImagePreprocessorports PIL's antialiased bilinear resize.ImageSortCheck: zero-shot Oxford-IIIT Pets, 37 breed prompts. M5 Pro: 94.85% on the 3,669 test photos (Python Core ML 94.77%, PyTorch 94.74%), 202 photos/s.ImageSortDemo: photos fly into a bar chart made of photos, one row per breed, with the current photo's top five.Head-to-head on 7,349 photos (same photos and prompts, after model load): Core ML 36.3 s, 202 photos/s, 262 MB peak memory, 94.26%; transformers fp32 on MPS (batch 32) 102.2 s, 72 photos/s, 4.24 GB, 94.11%.
Conversion and benchmarks: FluidInference/mobius
models/emb/siglip2/coreml.🤖 Generated with Claude Code