From 6082826ff4f9e7e34e8a856577ef5ffe87f70974 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Mon, 21 Sep 2026 01:12:34 -0400 Subject: [PATCH 01/16] docs: model inventory and conversion candidates Documents CUA-S1-FORMS as shipped, the laya-multilingual Core ML port as converted but not wired in, and the open-weight models worth converting next (laya-browser, laya-typed-decisions, GLiNER, Florence-2) with sizes, licenses, evidence, demo ideas, and what was ruled out and why. --- Documentation/Models.md | 81 +++++++++++++++++++++++++++++++++++++++++ README.md | 6 +++ 2 files changed, 87 insertions(+) create mode 100644 Documentation/Models.md diff --git a/Documentation/Models.md b/Documentation/Models.md new file mode 100644 index 0000000..2b47ad6 --- /dev/null +++ b/Documentation/Models.md @@ -0,0 +1,81 @@ +# Models + +What FluidUse runs today, what is converted but not wired in, and the open-weight +candidates worth converting next. Sizes are fp16 on disk. Traction was checked on +2026-09-21; licenses were read from each model card. + +## Shipped + +| Model | Params / fp16 | License | Role in FluidUse | +| --- | ---: | --- | --- | +| [cua-ai/cua-s1-forms](https://huggingface.co/cua-ai/cua-s1-forms) → [FluidInference/cua-s1-forms-coreml](https://huggingface.co/FluidInference/cua-s1-forms-coreml) | 706K / 1.4 MB | MIT | Matches each form field to a value from the document: `fill`, `check`, `click`, or `skip`. ~1 ms per decision on the Neural Engine. | + +CUA-S1-FORMS is a one-pass option scorer (a "System One" model in TypeSafe's terms), not an +LLM. It is a byte-level 2-layer Transformer with the option-attention head from +[jevlike](https://github.com/vinnylarouge/jevlike), trained by Cua on 10,000 synthetic forms. +It does not read goals, write text, or reason about dropdown options; those are the gaps the +rest of this page is about. + +## Converted, not yet wired in + +| Model | Params / fp16 | License | Where | +| --- | ---: | --- | --- | +| [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) → [FluidInference/laya-coreml](https://huggingface.co/FluidInference/laya-coreml) | 322M / 644 MB | Apache-2.0 | `LayaManager` on FluidAudio branch `feat/laya-coreml` (PR #946). Four token buckets, 32 option slots. Answers `choice` / `score` / `noul` questions over a text state in one pass. | + +On macOS 26 and newer, the on-device Apple Intelligence model covers entity extraction, +question answers, and dropdown choice through `FoundationModelsAssistant` with no conversion +at all. The candidates below matter for macOS 14 and 15, and for the one thing Apple's model +does not do: choosing an element on a page. + +## Candidates to convert + +Under 1 GB, permissively licensed, untouched so far. Ordered by how much each adds. + +| Model | Params / fp16 | License | Purpose | Evidence | +| --- | ---: | --- | --- | --- | +| [cklxx/laya-browser](https://huggingface.co/cklxx/laya-browser) v10s | 322M / 644 MB | Apache-2.0 | Picks which element to act on and which operation (`CLICK`, `TYPE_TEXT`, `SELECT`, `DONE`) from a goal and the page's element table. The role TypeSafe's Jev plays in browser agents. | Element top-1 0.63 over ~45 candidates, operation accuracy 0.88, 50% on 16 live browser tasks. Same architecture as the laya port above, so it is a weight bucket plus a sequence format, not a new conversion. | +| [convaiinnovations/laya-typed-decisions](https://huggingface.co/convaiinnovations/laya-typed-decisions) | 421M / 843 MB | Apache-2.0 | Dropdown choice, yes/no verification, and ordered-scale questions over a profile. | 0.766 on the typed-decisions benchmark, above Jev 1.13's published 0.727. Loads through `LayaManager`. | +| [urchade/gliner_small-v2.1](https://huggingface.co/urchade/gliner_small-v2.1) | ~150M / ~300 MB | Apache-2.0 | Pulls name, phone, employer, and any other labeled span out of an unstructured résumé or letter, replacing the `Label: value` parser. | 19k downloads a month; GLiNER repo 3.9k stars. | +| [microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) | 231M / 463 MB | MIT | Captions icons and regions in a screenshot, for apps with no accessibility tree. OmniParser's captioner. | 3M downloads a month. Only useful once a screenshot path exists. | + +Fallbacks, also under 1 GB and untouched: + +| Model | Params / fp16 | License | Purpose | +| --- | ---: | --- | --- | +| [urchade/gliner_multi_pii-v1](https://huggingface.co/urchade/gliner_multi_pii-v1) | ~300M / ~580 MB | Apache-2.0 | Same extractor, tuned for identity documents and personal-data fields. | +| [torontodeveloper/mind2web-candidate-ranker](https://huggingface.co/torontodeveloper/mind2web-candidate-ranker) | 184M / 369 MB | MIT | Ranks page elements by how likely each is the next click target; no operation choice. Recall@10 93% on unseen Mind2Web domains. | +| [shreyanbr/system-one-distilled](https://huggingface.co/shreyanbr/system-one-distilled) | 71M / 142 MB | Apache-2.0 | Smallest generic typed-decision model, for routing and gating where laya is overkill. | + +### More S1-class specialists + +Nothing else at CUA-S1-FORMS's scale exists with weights. The way to get more is to train +them: Cua's trainer, synthetic generator, and evaluator ship under MIT in +[trycua/cua `libs/cua-s1`](https://github.com/trycua/cua/tree/main/libs/cua-s1), and +[jevbetter](https://github.com/olanotolu/jevbetter) accepts the same JSONL with a stronger +encoder. Closed-vocabulary tasks that fit the shape, each 1 to 5 MB: + +- Submit-or-advance: which button submits, continues, or dismisses a dialog. +- Field-role tagging: search box, login form, cookie-banner accept. +- Clutter-or-ad per element. + +## Demo ideas + +| Model | Demo | What the viewer sees | +| --- | --- | --- | +| laya-browser | Goal-driven browsing in Safari: "book the cheapest nonstop to New York in October" | Each pick and its probability logged while the agent steps through Google Flights, fully offline. | +| laya-typed-decisions | Dropdowns and consent fields in an HR onboarding flow | "Years of experience", "work authorization", and "I agree" resolved from the profile with a confidence next to each. | +| gliner_small-v2.1 | Drop a real résumé PDF instead of a `Label: value` profile | The entity list fills from prose in under a second, then S1 fills the form from it. | +| gliner_multi_pii-v1 | Photo of an ID or insurance card via Vision OCR | Name, date of birth, member ID pulled from the card into a patient-intake form. Nothing leaves the machine. | +| Florence-2-base | Screenshot of an app with no accessibility tree | Icons and buttons labeled in place, then a decision made on those labels. | +| mind2web-candidate-ranker | "Where would you click next" overlay | Top five elements highlighted with scores, no action taken. | +| system-one-distilled | Routing benchmark | Hundreds of tickets classified per second with latency per call. | + +## Ruled out + +- **TypeSafe Jev**: hosted API only, no weights. +- **Screenshot grounders at 2B and up** (GUI-Owl-1.5-2B, UI-TARS-2B, ShowUI-2B, GUI-Actor-2B, Holo2-4B): 4 to 9 GB fp16, no Core ML precedent for their vision towers. +- **Non-commercial licenses**: Holo1.5-3B (Qwen Research), pngwn/system-one-qwen3.5-4b-scorer (CC-BY-NC), laya-vision-smolvlm and laya-grounded (CC-BY-NC). +- **AGPL**: OmniParser `icon_detect`, macpaw-research/yolov11l-ui-elements-detection. +- **No weights**: Apple Ferret-UI Lite, Ferret-UI 2, UI-JEPA. +- **"1 MB" heads that need a 0.6B backbone**: samatv256/mini-Jev and similar. +- **Decoder-based Jev clones over 1 GB**: Mapika/decider-0.8b and 2b, AndeyTait/JevForge-0.8B, IamBusy/OpenJev-0.6B. decider-2b is the best supported of these (20k downloads a month) if a larger chooser is ever needed on Mac. diff --git a/README.md b/README.md index f1cba4a..ebcd8f2 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,12 @@ reason about dropdown options, or write free text. The harness handles observation, typing, selection, and an answer sheet; uploads and essays are left to the person. Submit is never clicked unless enabled. +## Models + +[Documentation/Models.md](Documentation/Models.md) lists what runs today, what is converted +but not wired in, and the open-weight candidates worth converting next, with sizes, +licenses, and demo ideas. + ## License Apache 2.0. CUA-S1-FORMS is MIT, from Cua. From 2ab4590fbc88992478662881fa41ae2f449d3cab Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:02:06 -0400 Subject: [PATCH 02/16] docs: add decision game demos and sub-1B model candidates --- Documentation/GameDemos.md | 58 ++++++++++++++++++++++++++++++++++++++ Documentation/Models.md | 43 +++++++++++++++++++++++++++- README.md | 6 ++-- 3 files changed, 104 insertions(+), 3 deletions(-) create mode 100644 Documentation/GameDemos.md diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md new file mode 100644 index 0000000..a8c7140 --- /dev/null +++ b/Documentation/GameDemos.md @@ -0,0 +1,58 @@ +# Decision game demos + +Ideas for comparing small decision models in observable, repeatable environments. +These are proposed demos, not games currently shipped in FluidUse. Tetris, 2048, +and Snake are the starting examples from the discussion; Flappy Bird is the next +demo to build. The [model inventory](Models.md) identifies possible sub-1B models +and distinguishes text-state models from vision models. + +## Demo list + +| Priority | Game | Model's decision | What it reveals | Suggested state | +| --- | --- | --- | --- | --- | +| Next | **Flappy Bird** | `FLAP` or `COAST` each decision tick. | Reaction timing, latency, and costly one-step mistakes. | Bird height and velocity; next pipe distance and gap, or a rendered frame for a vision model. | +| Next | **Sokoban / Boxoban** | Pick a legal move or push. | Planning and irreversible traps. | Grid, player, crates, goals, and legal moves. | +| Next | **Minesweeper** | Open or flag a cell. | Decision-making under uncertainty. | Revealed grid and legal cells; show mine probabilities only if actually computed. | +| Later | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | +| Later | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | +| Later | **Hanabi** | Play, discard, or give a legal hint. | Cooperation with incomplete information. | Only what the acting player may observe. | +| Later | **MiniWoB-style web tasks** | Choose an element and operation. | Practical computer use with a clear success condition. | Accessibility element table and task goal; screenshots for vision models. | +| Later | **Breakout / Pong** | Move paddle left, right, or stay. | Ball prediction and control latency. | Ball/paddle positions and velocities. | +| Later | **Space Invaders** | Move, fire, or combine actions. | Avoidance and aiming under continuous pressure. | Player, projectiles, enemies, and cooldowns. | +| Later | **Racer** | Steer left, right, or straight. | Lookahead and fast corrections. | Track geometry, position, speed, and obstacles. | +| Existing example | **Tetris** | Choose a placement or movement. | Long-term board management. | Board, current piece, next piece if allowed, and legal placements. | +| Existing example | **2048** | Choose a legal slide. | Repeated choices with random future tiles. | Board and each legal resulting board before the random spawn. | +| Existing example | **Snake** | Choose direction. | Path planning while avoiding self-traps. | Grid, body, food, and current direction. | + +Flappy Bird is the clearest first comparison: two actions, immediate outcomes, +and a visible cost when a decision arrives late. Sokoban adds planning, and +Minesweeper adds uncertainty. Together, those three test different strengths +than the existing Tetris, 2048, and Snake examples. + +## Flappy Bird comparison specification + +1. The game engine advances at a fixed rate. Models receive a state at the same + decision interval and choose `FLAP` or `COAST`. The engine owns physics and + collision rules. +2. Run each model separately on the same seeded pipe sequences. Fix game speed, + decision interval, state schema, action semantics, and deadline. Give vision + models the same rendered frame; give text models the same structured facts. + Report vision and text-state results as different input tracks. +3. Show each action and its probability when available, plus the time it took. + An action probability expresses the model's preference between `FLAP` and + `COAST`; it is not a survival probability. +4. Keep the clock running during inference. If a result misses the deadline, + apply a declared fallback action and count a missed deadline. +5. Report pipes passed, survival ticks, median and p95 decision latency, missed + deadlines, and number of calls for every seed. Add a random policy and a + simple physics controller as reference runs. Publish the seed list and + per-run results before reporting aggregates. + +The [Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) +measures a frozen panel of static decision tasks. Its current score excludes +interactive environments, including Boxoban, Hanabi, Codenames, and MiniWoB++. +Use matched-seed game results to establish game performance; Index rank is only +a way to find candidate models. [PlayJev](https://github.com/OmniJev/PlayJev) +is a separate 0.8B vision model with published results on several of these games, +including Flappy Bird. Its published numbers use its own harness and should not +be placed on the same chart as new runs until the harness matches. diff --git a/Documentation/Models.md b/Documentation/Models.md index 2b47ad6..d31f35f 100644 --- a/Documentation/Models.md +++ b/Documentation/Models.md @@ -2,7 +2,9 @@ What FluidUse runs today, what is converted but not wired in, and the open-weight candidates worth converting next. Sizes are fp16 on disk. Traction was checked on -2026-09-21; licenses were read from each model card. +2026-09-21; licenses were read from each model card. The [Decision Index 0.1](https://huggingface.co/spaces/multimodalart/jev-decision-index) +snapshot below was checked on 2026-09-22. Its scores measure a frozen set of static +decision tasks, not game play. ## Shipped @@ -70,6 +72,45 @@ encoder. Closed-vocabulary tasks that fit the shape, each 1 to 5 MB: | mind2web-candidate-ranker | "Where would you click next" overlay | Top five elements highlighted with scores, no action taken. | | system-one-distilled | Routing benchmark | Hundreds of tickets classified per second with latency per call. | +### Game decisions + +See [Decision game demos](GameDemos.md) for the game list, the Flappy Bird +comparison specification, and the matched-seed evaluation protocol. + +### Promising Index models below 1B parameters + +These are **parameter-count** candidates from the [Index data](https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/main/data/index.json), +not models already converted for FluidUse. The Index score is its balanced static +score out of 100; it is included to identify candidates, not to predict game +performance. License and input type come from the linked model cards. Approximate +fp16 sizes include only weights (2 bytes per parameter), before runtime overhead. + +| Model | Params / fp16 | Index score | Input and best demo role | Status | +| --- | ---: | ---: | --- | --- | +| [Kev-0.6B](https://huggingface.co/jaredpalmer/kev-0.6b) | 596M / ~1.19 GB | 31.30 | Text state, typed `choice` / `score` / `noul`; first generic Flappy Bird decision candidate. | Apache-2.0; LoRA and pointer head require its Qwen3 base and a new runtime port. | +| [LFM2.5-350M-RLCD](https://huggingface.co/notnotsamuel/LFM2.5-350M-RLCD) | 354M / ~709 MB | 25.79 | Text decision baseline for Flappy Bird or routing. | `lfm1.0` custom license: review terms before conversion or redistribution. | +| [GLiNER2.5-base](https://huggingface.co/fastino/gliner2.5-base-v1) | 194M / ~387 MB | 24.70 | Text extraction and classification; best fit for form/document decisions. | Apache-2.0; needs a separate GLiNER runtime, not a direct game controller. | +| [GLiNER2.5-small](https://huggingface.co/fastino/gliner2.5-small-v1) | 74M / ~148 MB | 23.93 | Smaller text extraction/classification baseline. | Apache-2.0; different from the GLiNER small v2.1 candidate above. | +| [GLiNER2.5-multi](https://huggingface.co/fastino/gliner2.5-multi-v1) | 287M / ~575 MB | 22.42 | Multilingual form/document decisions. | Apache-2.0; separate GLiNER runtime. | +| [Decision-1.0-Lex](https://huggingface.co/llm-semantic-router/decision-1.0-lex) | 308M served in Index / ~616 MB | 19.57 | Text choices and scores; operational tasks and a Flappy Bird text-state trial. | Apache-2.0; card describes a 572M full checkpoint, so verify the artifact before sizing a port. | +| [Decision-1.0-Kai](https://huggingface.co/llm-semantic-router/decision-1.0-kai) | 308M served in Index / ~616 MB | 18.37 | General text decision comparator for Lex. | Apache-2.0; card calls it 0.6B, so confirm the exact artifact before conversion. | +| [Laya](https://huggingface.co/convaiinnovations/laya) | 421M / ~843 MB | 16.39 | Text typed decisions; comparison with the existing multilingual Core ML port. | Apache-2.0; the Index tests the English checkpoint, not `laya-multilingual`. | + +The Index also includes [NanoJev](https://huggingface.co/C-Tianyu/NanoJev) +(596M, 26.19), but its Hub card does not declare a license; keep it out of the +conversion shortlist until terms are clear. [Kev-0.5B](https://huggingface.co/jaredpalmer/kev-0.5b) +(494M, 30.34) is explicitly described by its author as a superseded prototype. +Neither is a first-choice port. All Index models in this table take text input; +there is no evidence in the Index that they can read Flappy Bird screenshots. + +[PlayJev-0.8B](https://huggingface.co/OmniJev/PlayJev-0.8B) is a separate, +Apache-2.0 **vision** candidate under 1B parameters, not a scored Index entrant. +It has a published Flappy Bird run and can take rendered frames, making it the +most direct visual comparator. Its fp16 weights are roughly 1.6 GB, above the +1 GB fp16 limit used for the main FluidUse conversion shortlist. Its published +Flappy Bird score must not be compared directly with a new harness run; use the +same physics, seeds, action timing, and scoring for every model in the demo. + ## Ruled out - **TypeSafe Jev**: hosted API only, no weights. diff --git a/README.md b/README.md index ebcd8f2..3505f6d 100644 --- a/README.md +++ b/README.md @@ -63,8 +63,10 @@ left to the person. Submit is never clicked unless enabled. ## Models [Documentation/Models.md](Documentation/Models.md) lists what runs today, what is converted -but not wired in, and the open-weight candidates worth converting next, with sizes, -licenses, and demo ideas. +but not wired in, and the open-weight candidates worth converting next, including +sub-1B-parameter candidates from the Jev Decision Index. +[Documentation/GameDemos.md](Documentation/GameDemos.md) lists game demos and specifies +a matched-seed Flappy Bird model comparison. ## License From bbee7296be467aae97de941eec581a8bda3f4a2b Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:03:48 -0400 Subject: [PATCH 03/16] docs: add Mario and chess demo concepts --- Documentation/GameDemos.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index a8c7140..df045fe 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -12,6 +12,8 @@ and distinguishes text-state models from vision models. | --- | --- | --- | --- | --- | | Next | **Flappy Bird** | `FLAP` or `COAST` each decision tick. | Reaction timing, latency, and costly one-step mistakes. | Bird height and velocity; next pipe distance and gap, or a rendered frame for a vision model. | | Next | **Sokoban / Boxoban** | Pick a legal move or push. | Planning and irreversible traps. | Grid, player, crates, goals, and legal moves. | +| Next | **Mario-style platformer** | Choose left, right, jump, or a legal combination at each tick. | Jump timing, momentum, obstacle anticipation, and longer action sequences. | Player position and velocity, nearby platforms, enemies, and camera offset; or a rendered frame. | +| Next | **Chess** | Choose or rank legal moves. | Tactical judgment, position evaluation, and planning across turns. | FEN, side to move, legal moves, and remaining time; a board image only for vision models. | | Next | **Minesweeper** | Open or flag a cell. | Decision-making under uncertainty. | Revealed grid and legal cells; show mine probabilities only if actually computed. | | Later | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | | Later | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | @@ -29,6 +31,25 @@ and a visible cost when a decision arrives late. Sokoban adds planning, and Minesweeper adds uncertainty. Together, those three test different strengths than the existing Tetris, 2048, and Snake examples. +Mario is a strong follow-up to Flappy Bird: both expose late decisions, while a +platformer also requires direction and jump choices across several frames. +Track distance reached, obstacles cleared, deaths, completed levels, and decision +latency on the same seeded levels. Define how many frames each action lasts so a +slower model is not silently given more game time. PlayJev already includes an +[Infinite Mario](https://github.com/OmniJev/PlayJev) environment, which is useful +as a reference. For a new public demo, use original or clearly licensed art; +the PlayJev repository notes that the Mario sprites in its vendored game belong +to Nintendo, despite the game's code being under the Unlicense. + +Chess has two useful modes. A **position challenge** gives every model the same +FEN and legal move list, then compares its ranked moves with published reference +values. The Decision Index already includes this kind of static test as +[ChessBench](https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/main/data/index.json). +A **full-game demo** pairs models against the same opponent at the same clock +setting and starting positions. Record legal-move rate, move time, game result, +and engine evaluation loss per move. Supply legal moves from a chess rules engine +so the model is judged on choosing among them, not on formatting notation. + ## Flappy Bird comparison specification 1. The game engine advances at a fixed rate. Models receive a state at the same From 96322b403294340317a3b6dace5e80e6f5ff61f2 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:10:56 -0400 Subject: [PATCH 04/16] docs: expand decision game demo list --- Documentation/GameDemos.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index df045fe..99df6d5 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -15,9 +15,16 @@ and distinguishes text-state models from vision models. | Next | **Mario-style platformer** | Choose left, right, jump, or a legal combination at each tick. | Jump timing, momentum, obstacle anticipation, and longer action sequences. | Player position and velocity, nearby platforms, enemies, and camera offset; or a rendered frame. | | Next | **Chess** | Choose or rank legal moves. | Tactical judgment, position evaluation, and planning across turns. | FEN, side to move, legal moves, and remaining time; a board image only for vision models. | | Next | **Minesweeper** | Open or flag a cell. | Decision-making under uncertainty. | Revealed grid and legal cells; show mine probabilities only if actually computed. | +| Next | **Wordle-style word game** | Choose the next valid guess. | Information gathering versus an immediate attempt to solve. | Previous guesses, color feedback, and a fixed candidate-word list. | +| Next | **Frogger-style crossing** | Move up, down, left, right, or wait. | Timing around moving hazards and choosing safe windows. | Player, lane hazards, speeds, and goal positions; or a rendered frame. | +| Next | **Connect Four** | Choose a non-full column. | Short tactical lookahead with only a few legal actions. | Board, player to move, and legal columns. | | Later | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | | Later | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | | Later | **Hanabi** | Play, discard, or give a legal hint. | Cooperation with incomplete information. | Only what the acting player may observe. | +| Later | **MiniGrid DoorKey** | Turn, move, pick up, or open. | Multi-step planning when a key must be found before reaching the goal. | Partial grid observation, carried item, and door state. | +| Later | **FrozenLake** | Choose one of four directions. | Planning under uncertain movement when the lake is slippery. | Map, position, goal, holes, and known movement rules. | +| Later | **Battleship** | Choose an untried target square. | Search under hidden information and using feedback from earlier turns. | Hits, misses, sunk ships, and remaining legal squares. | +| Later | **Tower defense** | Place, upgrade, sell, or wait. | Resource allocation with delayed effects. | Map, waves, towers, budget, and legal placements. | | Later | **MiniWoB-style web tasks** | Choose an element and operation. | Practical computer use with a clear success condition. | Accessibility element table and task goal; screenshots for vision models. | | Later | **Breakout / Pong** | Move paddle left, right, or stay. | Ball prediction and control latency. | Ball/paddle positions and velocities. | | Later | **Space Invaders** | Move, fire, or combine actions. | Avoidance and aiming under continuous pressure. | Player, projectiles, enemies, and cooldowns. | @@ -50,6 +57,14 @@ setting and starting positions. Record legal-move rate, move time, game result, and engine evaluation loss per move. Supply legal moves from a chess rules engine so the model is judged on choosing among them, not on formatting notation. +Wordle and Battleship add information-gathering decisions; Frogger adds moving +hazards; Connect Four is a compact turn-based duel. For reusable planning +environments, [MiniGrid DoorKey](https://minigrid.farama.org/environments/minigrid/DoorKeyEnv/) +and [Gymnasium FrozenLake](https://gymnasium.farama.org/main/environments/toy_text/frozen_lake/) +provide configurable tasks with small discrete action spaces. Start each model +from the same puzzle or seeded map, and keep its observations limited to what +the game rules permit. + ## Flappy Bird comparison specification 1. The game engine advances at a fixed rate. Models receive a state at the same From 46c392efa6fe177e5e3684d77a9793f5303e9994 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:11:58 -0400 Subject: [PATCH 05/16] docs: add endless runner and reusable game sources --- Documentation/GameDemos.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index 99df6d5..6068bdb 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -17,6 +17,7 @@ and distinguishes text-state models from vision models. | Next | **Minesweeper** | Open or flag a cell. | Decision-making under uncertainty. | Revealed grid and legal cells; show mine probabilities only if actually computed. | | Next | **Wordle-style word game** | Choose the next valid guess. | Information gathering versus an immediate attempt to solve. | Previous guesses, color feedback, and a fixed candidate-word list. | | Next | **Frogger-style crossing** | Move up, down, left, right, or wait. | Timing around moving hazards and choosing safe windows. | Player, lane hazards, speeds, and goal positions; or a rendered frame. | +| Next | **Subway Surfers-style runner** | Switch lane, jump, slide, or hold. | Fast obstacle recognition, action timing, and choosing between survival and coins. | Current lane, speed, nearby obstacles and distances; or a rendered frame. | | Next | **Connect Four** | Choose a non-full column. | Short tactical lookahead with only a few legal actions. | Board, player to move, and legal columns. | | Later | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | | Later | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | @@ -65,6 +66,29 @@ provide configurable tasks with small discrete action spaces. Start each model from the same puzzle or seeded map, and keep its observations limited to what the game rules permit. +## Reuse existing games + +The goal is to adapt playable games, not rebuild their physics, visuals, and +controls. A demo adapter should expose `reset(seed)`, `observe()`, `legalActions()`, +`step(action)`, `score()`, and `done()`. It can call an existing game's input +functions or simulate its keyboard controls. Keep gameplay and assets upstream +where possible; record the upstream revision and changes to the adapter. + +| Source | Good for | Reuse notes | +| --- | --- | --- | +| [PlayJev game harness](https://github.com/OmniJev/PlayJev) | Tetris, Snake, Pac-Man, Racer, Space Invaders, Sokoban, Infinite Mario, Floppy Bird, Breakout, 2048. | Already exposes seeded `start`, `step`, `frame`, `score`, `done`, and actions. Each vendored game has its own license; check art and levels separately before publishing. | +| [Neon Cyberpunk Runner](https://github.com/markstent/runner) | First Subway Surfers-style demo. | MIT-licensed Three.js browser game with three lanes, jump, slide, seedable track generation, and game logic separated from rendering. Add a thin adapter for observations and model actions. | +| [Cave Runner](https://github.com/tope-olajide/cave-runner) | Alternate 3D runner. | MIT-licensed, but its online score path uses Netlify and PlanetScale; assess whether a local-only demo can bypass that path. | +| [MiniGrid](https://minigrid.farama.org/environments/minigrid/) and [Gymnasium](https://gymnasium.farama.org/main/environments/) | DoorKey, FrozenLake, and other compact decision tasks. | Existing reset/step environments; add a viewer and a model input adapter. | + +For the runner, use the same generated track seeds, action interval, and speed +curve for every model. Report distance, obstacles cleared, coins, collisions, +and missed decision deadlines. An unlicensed [Subway Surfers clone](https://github.com/eeshadutta/Subway-Surfers) +exists, but its repository does not declare a reuse license, so it is not the +recommended source for a published demo. Call the result an endless runner and +use the upstream game's own art and name unless rights to the Subway Surfers +branding and assets are available. + ## Flappy Bird comparison specification 1. The game engine advances at a fixed rate. Models receive a state at the same From bb5a44af09f60b40ee07533340cc41e51fe1c4a8 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:14:56 -0400 Subject: [PATCH 06/16] docs: make PlayJev the starting game harness --- Documentation/GameDemos.md | 18 ++++++++++++++++++ README.md | 4 +++- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index 6068bdb..ae8c46d 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -81,6 +81,24 @@ where possible; record the upstream revision and changes to the adapter. | [Cave Runner](https://github.com/tope-olajide/cave-runner) | Alternate 3D runner. | MIT-licensed, but its online score path uses Netlify and PlanetScale; assess whether a local-only demo can bypass that path. | | [MiniGrid](https://minigrid.farama.org/environments/minigrid/) and [Gymnasium](https://gymnasium.farama.org/main/environments/) | DoorKey, FrozenLake, and other compact decision tasks. | Existing reset/step environments; add a viewer and a model input adapter. | +### Start with PlayJev + +Use [OmniJev/PlayJev](https://github.com/OmniJev/PlayJev) as the first game +harness to evaluate, rather than implementing its ten games again. Its browser +hook provides `window.pj.start(seed)`, `step(action)`, `frame()`, `score()`, +`done()`, and an action list. The repository also includes random and teacher +policies, recorded runs, and a [0.8B vision model](https://huggingface.co/OmniJev/PlayJev-0.8B) +that can serve as a visual reference. Keep the game, seed, tick rate, and action +list fixed while swapping the decision provider. For text-only models, add a +state extractor for each game and report text-state and pixel-input results in +separate tracks. Re-run all compared models in the same harness; PlayJev's +published scores are background context, not directly comparable results. + +Before vendoring or publishing a game, review its own license and bundled art. +PlayJev documents the licenses of its ten vendored games and notes that some +sprites are owned by third parties. The runner above fills a gap in its current +game list; it is not one of PlayJev's ten games. + For the runner, use the same generated track seeds, action interval, and speed curve for every model. Report distance, obstacles cleared, coins, collisions, and missed decision deadlines. An unlicensed [Subway Surfers clone](https://github.com/eeshadutta/Subway-Surfers) diff --git a/README.md b/README.md index 3505f6d..f0396ed 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,9 @@ left to the person. Submit is never clicked unless enabled. but not wired in, and the open-weight candidates worth converting next, including sub-1B-parameter candidates from the Jev Decision Index. [Documentation/GameDemos.md](Documentation/GameDemos.md) lists game demos and specifies -a matched-seed Flappy Bird model comparison. +a matched-seed Flappy Bird model comparison. [PlayJev](https://github.com/OmniJev/PlayJev) +is the proposed starting harness for ten existing browser games; an open-source +three-lane runner is listed for the Subway Surfers-style demo. ## License From c48a0227ff73deb3ac25c8af281d0d3d948581ba Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:46:39 -0400 Subject: [PATCH 07/16] docs: rank decision game demos by build value --- Documentation/GameDemos.md | 78 ++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 21 deletions(-) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index ae8c46d..844e609 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -6,30 +6,66 @@ and Snake are the starting examples from the discussion; Flappy Bird is the next demo to build. The [model inventory](Models.md) identifies possible sub-1B models and distinguishes text-state models from vision models. +## Recommended build order + +This is a judgment about **new demo value for FluidUse**, not a measured model +ranking. It favors visible differences between models, a clear score, distinct +decision types, and reusable game code. Tetris, 2048, and Snake rank last only +because they are already the starting examples. [PlayJev](https://github.com/OmniJev/PlayJev) +reduces the work for ten browser games; the three-lane +[Neon Cyberpunk Runner](https://github.com/markstent/runner) is the reuse path +for a Subway Surfers-style demo. + +| Rank | Game | Main reason to demo it | +| ---: | --- | --- | +| 1 | Flappy Bird | Two choices make latency and model differences immediately visible. | +| 2 | Sokoban / Boxoban | Clear planning failures; reusable PlayJev game. | +| 3 | Subway Surfers-style runner | High visual appeal and distinct lane, jump, and slide choices; reusable runner exists. | +| 4 | Mario-style platformer | Familiar, varied actions and long action sequences; PlayJev has a game harness. | +| 5 | Chess | Recognizable tactics and a static-position mode that is easy to compare. | +| 6 | Wordle-style word game | Adds information-gathering decisions with a compact state. | +| 7 | Minesweeper | Shows risk-sensitive choices and uncertainty. | +| 8 | Frogger-style crossing | Shows timing around moving hazards with simple actions. | +| 9 | Connect Four | Small, legal move set and easy human-versus-model play. | +| 10 | Pac-Man | Visually engaging risk/reward decisions; PlayJev has a game harness. | +| 11 | Space Invaders | Fast aiming and dodging; PlayJev has a game harness. | +| 12 | MiniWoB-style web tasks | Closest to FluidUse's computer-use purpose, with objective task success. | +| 13 | Breakout / Pong | Simple control and visible prediction errors; PlayJev includes Breakout. | +| 14 | MiniGrid DoorKey | Multi-step planning with an existing environment. | +| 15 | Codenames | Semantic association, distinct from movement games. | +| 16 | Racer | Fast steering decisions; PlayJev has a game harness. | +| 17 | Battleship | Hidden-information search, but slower to watch. | +| 18 | Hanabi | Cooperative partial information, but harder to explain at a glance. | +| 19 | Tower defense | Resource allocation, but a large action and state space. | +| 20 | FrozenLake | Useful stochastic baseline, but visually less compelling. | +| 21 | Tetris | Already a starting example. | +| 22 | 2048 | Already a starting example. | +| 23 | Snake | Already a starting example. | + ## Demo list -| Priority | Game | Model's decision | What it reveals | Suggested state | +| Status | Game | Model's decision | What it reveals | Suggested state | | --- | --- | --- | --- | --- | -| Next | **Flappy Bird** | `FLAP` or `COAST` each decision tick. | Reaction timing, latency, and costly one-step mistakes. | Bird height and velocity; next pipe distance and gap, or a rendered frame for a vision model. | -| Next | **Sokoban / Boxoban** | Pick a legal move or push. | Planning and irreversible traps. | Grid, player, crates, goals, and legal moves. | -| Next | **Mario-style platformer** | Choose left, right, jump, or a legal combination at each tick. | Jump timing, momentum, obstacle anticipation, and longer action sequences. | Player position and velocity, nearby platforms, enemies, and camera offset; or a rendered frame. | -| Next | **Chess** | Choose or rank legal moves. | Tactical judgment, position evaluation, and planning across turns. | FEN, side to move, legal moves, and remaining time; a board image only for vision models. | -| Next | **Minesweeper** | Open or flag a cell. | Decision-making under uncertainty. | Revealed grid and legal cells; show mine probabilities only if actually computed. | -| Next | **Wordle-style word game** | Choose the next valid guess. | Information gathering versus an immediate attempt to solve. | Previous guesses, color feedback, and a fixed candidate-word list. | -| Next | **Frogger-style crossing** | Move up, down, left, right, or wait. | Timing around moving hazards and choosing safe windows. | Player, lane hazards, speeds, and goal positions; or a rendered frame. | -| Next | **Subway Surfers-style runner** | Switch lane, jump, slide, or hold. | Fast obstacle recognition, action timing, and choosing between survival and coins. | Current lane, speed, nearby obstacles and distances; or a rendered frame. | -| Next | **Connect Four** | Choose a non-full column. | Short tactical lookahead with only a few legal actions. | Board, player to move, and legal columns. | -| Later | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | -| Later | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | -| Later | **Hanabi** | Play, discard, or give a legal hint. | Cooperation with incomplete information. | Only what the acting player may observe. | -| Later | **MiniGrid DoorKey** | Turn, move, pick up, or open. | Multi-step planning when a key must be found before reaching the goal. | Partial grid observation, carried item, and door state. | -| Later | **FrozenLake** | Choose one of four directions. | Planning under uncertain movement when the lake is slippery. | Map, position, goal, holes, and known movement rules. | -| Later | **Battleship** | Choose an untried target square. | Search under hidden information and using feedback from earlier turns. | Hits, misses, sunk ships, and remaining legal squares. | -| Later | **Tower defense** | Place, upgrade, sell, or wait. | Resource allocation with delayed effects. | Map, waves, towers, budget, and legal placements. | -| Later | **MiniWoB-style web tasks** | Choose an element and operation. | Practical computer use with a clear success condition. | Accessibility element table and task goal; screenshots for vision models. | -| Later | **Breakout / Pong** | Move paddle left, right, or stay. | Ball prediction and control latency. | Ball/paddle positions and velocities. | -| Later | **Space Invaders** | Move, fire, or combine actions. | Avoidance and aiming under continuous pressure. | Player, projectiles, enemies, and cooldowns. | -| Later | **Racer** | Steer left, right, or straight. | Lookahead and fast corrections. | Track geometry, position, speed, and obstacles. | +| Candidate | **Flappy Bird** | `FLAP` or `COAST` each decision tick. | Reaction timing, latency, and costly one-step mistakes. | Bird height and velocity; next pipe distance and gap, or a rendered frame for a vision model. | +| Candidate | **Sokoban / Boxoban** | Pick a legal move or push. | Planning and irreversible traps. | Grid, player, crates, goals, and legal moves. | +| Candidate | **Mario-style platformer** | Choose left, right, jump, or a legal combination at each tick. | Jump timing, momentum, obstacle anticipation, and longer action sequences. | Player position and velocity, nearby platforms, enemies, and camera offset; or a rendered frame. | +| Candidate | **Chess** | Choose or rank legal moves. | Tactical judgment, position evaluation, and planning across turns. | FEN, side to move, legal moves, and remaining time; a board image only for vision models. | +| Candidate | **Minesweeper** | Open or flag a cell. | Decision-making under uncertainty. | Revealed grid and legal cells; show mine probabilities only if actually computed. | +| Candidate | **Wordle-style word game** | Choose the next valid guess. | Information gathering versus an immediate attempt to solve. | Previous guesses, color feedback, and a fixed candidate-word list. | +| Candidate | **Frogger-style crossing** | Move up, down, left, right, or wait. | Timing around moving hazards and choosing safe windows. | Player, lane hazards, speeds, and goal positions; or a rendered frame. | +| Candidate | **Subway Surfers-style runner** | Switch lane, jump, slide, or hold. | Fast obstacle recognition, action timing, and choosing between survival and coins. | Current lane, speed, nearby obstacles and distances; or a rendered frame. | +| Candidate | **Connect Four** | Choose a non-full column. | Short tactical lookahead with only a few legal actions. | Board, player to move, and legal columns. | +| Candidate | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | +| Candidate | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | +| Candidate | **Hanabi** | Play, discard, or give a legal hint. | Cooperation with incomplete information. | Only what the acting player may observe. | +| Candidate | **MiniGrid DoorKey** | Turn, move, pick up, or open. | Multi-step planning when a key must be found before reaching the goal. | Partial grid observation, carried item, and door state. | +| Candidate | **FrozenLake** | Choose one of four directions. | Planning under uncertain movement when the lake is slippery. | Map, position, goal, holes, and known movement rules. | +| Candidate | **Battleship** | Choose an untried target square. | Search under hidden information and using feedback from earlier turns. | Hits, misses, sunk ships, and remaining legal squares. | +| Candidate | **Tower defense** | Place, upgrade, sell, or wait. | Resource allocation with delayed effects. | Map, waves, towers, budget, and legal placements. | +| Candidate | **MiniWoB-style web tasks** | Choose an element and operation. | Practical computer use with a clear success condition. | Accessibility element table and task goal; screenshots for vision models. | +| Candidate | **Breakout / Pong** | Move paddle left, right, or stay. | Ball prediction and control latency. | Ball/paddle positions and velocities. | +| Candidate | **Space Invaders** | Move, fire, or combine actions. | Avoidance and aiming under continuous pressure. | Player, projectiles, enemies, and cooldowns. | +| Candidate | **Racer** | Steer left, right, or straight. | Lookahead and fast corrections. | Track geometry, position, speed, and obstacles. | | Existing example | **Tetris** | Choose a placement or movement. | Long-term board management. | Board, current piece, next piece if allowed, and legal placements. | | Existing example | **2048** | Choose a legal slide. | Repeated choices with random future tiles. | Board and each legal resulting board before the random spawn. | | Existing example | **Snake** | Choose direction. | Path planning while avoiding self-traps. | Grid, body, food, and current direction. | From 175bad18db196da77888d1532625bd7bfc43fb83 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 15:58:21 -0400 Subject: [PATCH 08/16] docs: specify fair two-player game demos --- Documentation/GameDemos.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index 844e609..f63a842 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -94,6 +94,34 @@ setting and starting positions. Record legal-move rate, move time, game result, and engine evaluation loss per move. Supply legal moves from a chess rules engine so the model is judged on choosing among them, not on formatting notation. +## Two-player games + +Offer three viewer modes, but keep the benchmark mode separate from the show match: + +| Mode | Purpose | How it works | +| --- | --- | --- | +| Human vs model | Interactive demo. | The person picks a side and a legal move; show the model's top choices, probabilities when available, and response time. Do not count these self-selected games in model rankings. | +| Model vs fixed opponent | Primary comparison. | Each candidate faces the same version and settings of a reference policy. Run paired games from the same starting positions with colors or first turn swapped. | +| Model vs model | Spectator match. | Run a round robin with paired colors and matched opening positions. Show wins, draws, losses, illegal or timed-out decisions, and move time; do not infer a global ranking from one head-to-head game. | + +Start with **Connect Four**: it has a small action set, short games, and no clock +pressure. Use a rules engine to provide legal columns, then a fixed search policy +as the reference opponent. Add **chess** second. [chess.js](https://github.com/jhlywa/chess.js) +can provide legal moves, validation, and game-end detection in a browser demo; +[Stockfish](https://github.com/official-stockfish/Stockfish) can be the fixed +reference opponent at a stated strength and time budget. Models see the same FEN, +move history limit, legal move list, and clock information. Rotate colors and +use a fixed set of opening positions. Keep any engine evaluation out of the +model's input; calculate it afterward for the viewer. + +For hidden-information games, keep roles and observations honest. In Battleship, +each player sees only its own ships and prior shots. In Codenames, score clue +giver and guesser separately, with a fixed partner for comparable runs. Hanabi +is cooperative rather than adversarial: use fixed teammate policies and report +team score, not win rate against another model. Record opponent version, seeds, +starting positions, rules, action deadline, and every move so a result can be +replayed. + Wordle and Battleship add information-gathering decisions; Frogger adds moving hazards; Connect Four is a compact turn-based duel. For reusable planning environments, [MiniGrid DoorKey](https://minigrid.farama.org/environments/minigrid/DoorKeyEnv/) From 6e1572e86544042a3bc90d6e5ec2dd5ea4583a61 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 16:00:28 -0400 Subject: [PATCH 09/16] docs: prioritize model battles and self-play --- Documentation/GameDemos.md | 46 +++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md index f63a842..8a46792 100644 --- a/Documentation/GameDemos.md +++ b/Documentation/GameDemos.md @@ -8,9 +8,9 @@ and distinguishes text-state models from vision models. ## Recommended build order -This is a judgment about **new demo value for FluidUse**, not a measured model -ranking. It favors visible differences between models, a clear score, distinct -decision types, and reusable game code. Tetris, 2048, and Snake rank last only +This is a judgment about **solo and side-by-side demo value for FluidUse**, not +a measured model ranking. It favors visible differences between models, a clear +score, distinct decision types, and reusable game code. Tetris, 2048, and Snake rank last only because they are already the starting examples. [PlayJev](https://github.com/OmniJev/PlayJev) reduces the work for ten browser games; the three-lane [Neon Cyberpunk Runner](https://github.com/markstent/runner) is the reuse path @@ -55,6 +55,7 @@ for a Subway Surfers-style demo. | Candidate | **Frogger-style crossing** | Move up, down, left, right, or wait. | Timing around moving hazards and choosing safe windows. | Player, lane hazards, speeds, and goal positions; or a rendered frame. | | Candidate | **Subway Surfers-style runner** | Switch lane, jump, slide, or hold. | Fast obstacle recognition, action timing, and choosing between survival and coins. | Current lane, speed, nearby obstacles and distances; or a rendered frame. | | Candidate | **Connect Four** | Choose a non-full column. | Short tactical lookahead with only a few legal actions. | Board, player to move, and legal columns. | +| Candidate | **Multiplayer Snake arena** | Choose direction while several snakes move simultaneously. | Direct competition, collisions, and survival under pressure. | Board, all visible snakes, food, and legal directions. | | Candidate | **Pac-Man** | Direction at each junction. | Reward versus moving hazards. | Map, player, ghosts, pellets, and power timer. | | Candidate | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | | Candidate | **Hanabi** | Play, discard, or give a legal hint. | Cooperation with incomplete information. | Only what the acting player may observe. | @@ -89,31 +90,45 @@ Chess has two useful modes. A **position challenge** gives every model the same FEN and legal move list, then compares its ranked moves with published reference values. The Decision Index already includes this kind of static test as [ChessBench](https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/main/data/index.json). -A **full-game demo** pairs models against the same opponent at the same clock -setting and starting positions. Record legal-move rate, move time, game result, +A **full-game demo** pairs models against each other at the same clock setting +and matched starting positions. Record legal-move rate, move time, game result, and engine evaluation loss per move. Supply legal moves from a chess rules engine so the model is judged on choosing among them, not on formatting notation. -## Two-player games +## Competitive games -Offer three viewer modes, but keep the benchmark mode separate from the show match: +Make **model vs model** the main viewer mode. The following modes use the same +decision interface, with fixed-opponent runs retained as a diagnostic: | Mode | Purpose | How it works | | --- | --- | --- | -| Human vs model | Interactive demo. | The person picks a side and a legal move; show the model's top choices, probabilities when available, and response time. Do not count these self-selected games in model rankings. | -| Model vs fixed opponent | Primary comparison. | Each candidate faces the same version and settings of a reference policy. Run paired games from the same starting positions with colors or first turn swapped. | -| Model vs model | Spectator match. | Run a round robin with paired colors and matched opening positions. Show wins, draws, losses, illegal or timed-out decisions, and move time; do not infer a global ranking from one head-to-head game. | - -Start with **Connect Four**: it has a small action set, short games, and no clock -pressure. Use a rules engine to provide legal columns, then a fixed search policy -as the reference opponent. Add **chess** second. [chess.js](https://github.com/jhlywa/chess.js) +| Model vs model | Main spectator match. | Run a round robin with paired colors and matched opening positions. Show wins, draws, losses, illegal or timed-out decisions, and move time. | +| Same model vs itself | Self-play showcase. | Launch two independent instances of the same checkpoint. Swap colors across paired games and vary openings. If choices are sampled, give each instance an independent random seed and show the sampling setting. | +| Multi-model arena | Actual battle royale. | Put three or more models in one simultaneous-action game, such as multiplayer Snake. Give every model the same board snapshot each tick, collect actions independently, then resolve them together at the deadline. Rotate spawn positions across matches. | +| Human vs model | Interactive side mode. | The person picks a side and a legal move; show the model's top choices, probabilities when available, and response time. Do not count these self-selected games in model rankings. | +| Model vs fixed opponent | Diagnostic. | Each candidate faces the same version and settings of a reference policy. Run paired games from the same starting positions with colors or first turn swapped. | + +For competitive demos, start with **Connect Four** for two-player self-play and +cross-model matches, then **multiplayer Snake** for a true arena, then **chess**. +An existing [MIT-licensed multiplayer Snake game](https://github.com/simondiep/node-multiplayer-snake) +has spectator mode, bots, and adjustable speed; evaluate it as a reusable base +instead of expanding PlayJev's single-player Snake from scratch. Connect Four +has a small action set and short games. Use a rules engine to provide legal +columns and a fixed search policy only for diagnostics. [chess.js](https://github.com/jhlywa/chess.js) can provide legal moves, validation, and game-end detection in a browser demo; [Stockfish](https://github.com/official-stockfish/Stockfish) can be the fixed -reference opponent at a stated strength and time budget. Models see the same FEN, +reference opponent for diagnostics at a stated strength and time budget. Models see the same FEN, move history limit, legal move list, and clock information. Rotate colors and use a fixed set of opening positions. Keep any engine evaluation out of the model's input; calculate it afterward for the viewer. +In self-play, identical deterministic policies can repeat the same line or draw +often. Vary the opening positions and show that behavior honestly; optional +sampling makes matches more varied, but its temperature must be declared and +held fixed across models. For rankings, use many paired matches and record the +model version, side or spawn, opening or map seed, action deadline, and full +move log. A single spectacular match is a demo clip, not a performance estimate. + For hidden-information games, keep roles and observations honest. In Battleship, each player sees only its own ships and prior shots. In Codenames, score clue giver and guesser separately, with a fixed partner for comparable runs. Hanabi @@ -142,6 +157,7 @@ where possible; record the upstream revision and changes to the adapter. | --- | --- | --- | | [PlayJev game harness](https://github.com/OmniJev/PlayJev) | Tetris, Snake, Pac-Man, Racer, Space Invaders, Sokoban, Infinite Mario, Floppy Bird, Breakout, 2048. | Already exposes seeded `start`, `step`, `frame`, `score`, `done`, and actions. Each vendored game has its own license; check art and levels separately before publishing. | | [Neon Cyberpunk Runner](https://github.com/markstent/runner) | First Subway Surfers-style demo. | MIT-licensed Three.js browser game with three lanes, jump, slide, seedable track generation, and game logic separated from rendering. Add a thin adapter for observations and model actions. | +| [Node Multiplayer Snake](https://github.com/simondiep/node-multiplayer-snake) | Multi-model survival arena. | MIT-licensed browser game with spectator mode, bots, and adjustable speed. Adapt its player controllers to model actions and add seeded resets. | | [Cave Runner](https://github.com/tope-olajide/cave-runner) | Alternate 3D runner. | MIT-licensed, but its online score path uses Netlify and PlanetScale; assess whether a local-only demo can bypass that path. | | [MiniGrid](https://minigrid.farama.org/environments/minigrid/) and [Gymnasium](https://gymnasium.farama.org/main/environments/) | DoorKey, FrozenLake, and other compact decision tasks. | Existing reset/step environments; add a viewer and a model input adapter. | From ac58cf12a0f41f59e6bbdda1d668a4eb55787e4d Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 16:09:53 -0400 Subject: [PATCH 10/16] docs: inventory GLiClass Core ML and Hub assets --- Documentation/Models.md | 25 +++++++++---------------- README.md | 8 +++++--- 2 files changed, 14 insertions(+), 19 deletions(-) diff --git a/Documentation/Models.md b/Documentation/Models.md index d31f35f..c41962c 100644 --- a/Documentation/Models.md +++ b/Documentation/Models.md @@ -1,33 +1,26 @@ # Models -What FluidUse runs today, what is converted but not wired in, and the open-weight -candidates worth converting next. Sizes are fp16 on disk. Traction was checked on +What FluidUse runs today and the open-weight candidates worth converting next. +Sizes are fp16 on disk unless stated otherwise. Traction was checked on 2026-09-21; licenses were read from each model card. The [Decision Index 0.1](https://huggingface.co/spaces/multimodalart/jev-decision-index) snapshot below was checked on 2026-09-22. Its scores measure a frozen set of static decision tasks, not game play. -## Shipped +## Converted and integrated | Model | Params / fp16 | License | Role in FluidUse | | --- | ---: | --- | --- | | [cua-ai/cua-s1-forms](https://huggingface.co/cua-ai/cua-s1-forms) → [FluidInference/cua-s1-forms-coreml](https://huggingface.co/FluidInference/cua-s1-forms-coreml) | 706K / 1.4 MB | MIT | Matches each form field to a value from the document: `fill`, `check`, `click`, or `skip`. ~1 ms per decision on the Neural Engine. | +| [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) → [FluidInference/laya-coreml](https://huggingface.co/FluidInference/laya-coreml) | 322M / ~644 MB | Apache-2.0 | `LayaManager` answers text-state `choice`, `score`, and `noul` questions. Core ML buckets at 128/256/512/1024 tokens; Tetris and 2048 demos. | +| [knowledgator/gliclass-edge-v3.0](https://huggingface.co/knowledgator/gliclass-edge-v3.0) → [FluidInference/gliclass-edge-apps-coreml](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) | 32.7M / 65.7 MB (33.0 MB LUT8 L128) | Apache-2.0 | Application-tuned `GLiClassManager` compares 2–25 supplied labels in one pass; Tetris, 2048, and GLiClass-vs-Laya demos. FP16 buckets at 128/256/512 tokens. | CUA-S1-FORMS is a one-pass option scorer (a "System One" model in TypeSafe's terms), not an LLM. It is a byte-level 2-layer Transformer with the option-attention head from [jevlike](https://github.com/vinnylarouge/jevlike), trained by Cua on 10,000 synthetic forms. -It does not read goals, write text, or reason about dropdown options; those are the gaps the -rest of this page is about. - -## Converted, not yet wired in - -| Model | Params / fp16 | License | Where | -| --- | ---: | --- | --- | -| [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) → [FluidInference/laya-coreml](https://huggingface.co/FluidInference/laya-coreml) | 322M / 644 MB | Apache-2.0 | `LayaManager` on FluidAudio branch `feat/laya-coreml` (PR #946). Four token buckets, 32 option slots. Answers `choice` / `score` / `noul` questions over a text state in one pass. | - -On macOS 26 and newer, the on-device Apple Intelligence model covers entity extraction, -question answers, and dropdown choice through `FoundationModelsAssistant` with no conversion -at all. The candidates below matter for macOS 14 and 15, and for the one thing Apple's model -does not do: choosing an element on a page. +It does not read goals, write text, or reason about dropdown options. Laya and +GLiClass handle more general decisions but the form-filling app still uses CUA-S1-FORMS. +Apple's built-in Foundation Models are not a FluidUse conversion and are not +included in this inventory. ## Candidates to convert diff --git a/README.md b/README.md index 959bc56..583c5f9 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,9 @@ The on-device models, measured on the same Mac with checked-in reports: [Benchma CUA-S1-FORMS: 0.9 ms per decision on the Neural Engine, accuracy identical to PyTorch on the 24,370-row synthetic test. laya: 3.6 ms per short question, identical to PyTorch on laya's ten published suites, e8 buckets 30% smaller at the same accuracy. GLiClass Edge Apps v2: 1.61 ms FP16 -or 1.81 ms LUT8 for a two-option L128 decision, with its conversion pipeline in +or 1.81 ms LUT8 for a two-option L128 decision. Its +[Core ML packages and config](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) +are on Hugging Face, with the conversion pipeline in [mobius PR #101](https://github.com/FluidInference/mobius/pull/101). ## Scope @@ -133,8 +135,8 @@ left to the person. Submit is never clicked unless enabled. ## Models -[Documentation/Models.md](Documentation/Models.md) lists what runs today, what is converted -but not wired in, and the open-weight candidates worth converting next, including +[Documentation/Models.md](Documentation/Models.md) lists the three converted models and +open-weight candidates worth converting next, including sub-1B-parameter candidates from the Jev Decision Index. [Documentation/GameDemos.md](Documentation/GameDemos.md) lists game demos and specifies a matched-seed Flappy Bird model comparison. [PlayJev](https://github.com/OmniJev/PlayJev) From 7599799e56027be36b2b369a4150fc30f79eebce Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Tue, 22 Sep 2026 16:17:44 -0400 Subject: [PATCH 11/16] Load GLiClass Core ML buckets from published Hub config --- Documentation/Models.md | 2 +- README.md | 7 +- .../Decision2048BenchModel.swift | 14 +- Sources/Decision2048BenchDemo/README.md | 5 +- .../FluidUse/GLiClass/GLiClassManager.swift | 13 ++ .../GLiClass/GLiClassModelStore.swift | 154 ++++++++++++++++++ Sources/FluidUseLaya/Game2048Command.swift | 11 +- Sources/FluidUseLaya/LayaTetrisCommand.swift | 16 +- Sources/GLiClass2048Demo/Game2048Model.swift | 16 +- Sources/GLiClass2048Demo/README.md | 7 +- Sources/LayaTetrisDemo/GameModel.swift | 17 +- Sources/LayaTetrisDemo/README.md | 11 +- .../Fixtures/gliclass-hub-config.json | 46 ++++++ .../GLiClassModelStoreTests.swift | 33 ++++ 14 files changed, 311 insertions(+), 41 deletions(-) create mode 100644 Sources/FluidUse/GLiClass/GLiClassModelStore.swift create mode 100644 Tests/FluidUseTests/Fixtures/gliclass-hub-config.json create mode 100644 Tests/FluidUseTests/GLiClassModelStoreTests.swift diff --git a/Documentation/Models.md b/Documentation/Models.md index c41962c..55989b1 100644 --- a/Documentation/Models.md +++ b/Documentation/Models.md @@ -12,7 +12,7 @@ decision tasks, not game play. | --- | ---: | --- | --- | | [cua-ai/cua-s1-forms](https://huggingface.co/cua-ai/cua-s1-forms) → [FluidInference/cua-s1-forms-coreml](https://huggingface.co/FluidInference/cua-s1-forms-coreml) | 706K / 1.4 MB | MIT | Matches each form field to a value from the document: `fill`, `check`, `click`, or `skip`. ~1 ms per decision on the Neural Engine. | | [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) → [FluidInference/laya-coreml](https://huggingface.co/FluidInference/laya-coreml) | 322M / ~644 MB | Apache-2.0 | `LayaManager` answers text-state `choice`, `score`, and `noul` questions. Core ML buckets at 128/256/512/1024 tokens; Tetris and 2048 demos. | -| [knowledgator/gliclass-edge-v3.0](https://huggingface.co/knowledgator/gliclass-edge-v3.0) → [FluidInference/gliclass-edge-apps-coreml](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) | 32.7M / 65.7 MB (33.0 MB LUT8 L128) | Apache-2.0 | Application-tuned `GLiClassManager` compares 2–25 supplied labels in one pass; Tetris, 2048, and GLiClass-vs-Laya demos. FP16 buckets at 128/256/512 tokens. | +| [knowledgator/gliclass-edge-v3.0](https://huggingface.co/knowledgator/gliclass-edge-v3.0) → [FluidInference/gliclass-edge-apps-coreml](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) | 32.7M / 65.7 MB (33.0 MB LUT8 L128) | Apache-2.0 | Application-tuned `GLiClassManager` compares 2–25 supplied labels in one pass; Tetris, 2048, and GLiClass-vs-Laya demos. Its Hub `config.json` drives automatic downloads of FP16 buckets at 128/256/512 tokens or LUT8 at 128 tokens. | CUA-S1-FORMS is a one-pass option scorer (a "System One" model in TypeSafe's terms), not an LLM. It is a byte-level 2-layer Transformer with the option-attention head from diff --git a/README.md b/README.md index 583c5f9..7a4f980 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ questions, reference answers, reports and conversion live in swift run -c release FluidUseLaya answer --state "…" --type choice \ --instructions "What does the customer want?" --options "refund|order status|technical help" swift run -c release FluidUseLaya tetris --shortlist --describe graded --pieces 200 # headless Tetris, P(clean) per landing -swift run -c release FluidUseLaya 2048 --model-dir /path/to/gliclass --precision lut8 --games 10 +swift run -c release FluidUseLaya 2048 --precision lut8 --games 10 swift run -c release FluidUseLaya benchmark --suites /benchmark/suites.jsonl --reference /benchmark/reference-rows.jsonl swift run -c release LayaTetrisDemo # SwiftUI: GLiClass/laya play Tetris swift run -c release GLiClass2048Demo # SwiftUI: GLiClass plays 2048 @@ -125,6 +125,11 @@ or 1.81 ms LUT8 for a two-option L128 decision. Its [Core ML packages and config](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) are on Hugging Face, with the conversion pipeline in [mobius PR #101](https://github.com/FluidInference/mobius/pull/101). +GLiClass demos download the selected Core ML bucket and tokenizer on first use. The loader reads the +published `config.json`, checks each file against `checksums.json`, and caches the assets under +`~/Library/Application Support/FluidUse/Models/gliclass-edge-apps-coreml`. `GLICLASS_MODEL_DIR` +or CLI `--model-dir` still selects a local directory. The Hub publishes FP16 at L128/L256/L512 and +LUT8 at L128; other local precision variants require an explicit local directory. ## Scope diff --git a/Sources/Decision2048BenchDemo/Decision2048BenchModel.swift b/Sources/Decision2048BenchDemo/Decision2048BenchModel.swift index 04e6965..e1ccd0e 100644 --- a/Sources/Decision2048BenchDemo/Decision2048BenchModel.swift +++ b/Sources/Decision2048BenchDemo/Decision2048BenchModel.swift @@ -94,13 +94,15 @@ final class Decision2048BenchModel: ObservableObject { Task { do { let environment = ProcessInfo.processInfo.environment - guard let gliClassDirectory = environment["GLICLASS_MODEL_DIR"], !gliClassDirectory.isEmpty else { - throw GLiClassError.invalidAsset("Set GLICLASS_MODEL_DIR for the 2048 benchmark") + let gliClassConfiguration = GLiClassManager.Configuration( + lengths: [128], precision: environment["GLICLASS_PRECISION"] ?? "lut8") + let loadedGLiClass: GLiClassManager + if let directory = environment["GLICLASS_MODEL_DIR"], !directory.isEmpty { + loadedGLiClass = try await GLiClassManager.load( + from: URL(fileURLWithPath: directory), configuration: gliClassConfiguration) + } else { + loadedGLiClass = try await GLiClassManager.load(configuration: gliClassConfiguration) } - let loadedGLiClass = try await GLiClassManager.load( - from: URL(fileURLWithPath: gliClassDirectory), - configuration: .init( - lengths: [128], precision: environment["GLICLASS_PRECISION"] ?? "lut8")) loadStatus = "Loading Laya E8…" let layaConfiguration = LayaManager.Configuration( lengths: [128], precision: environment["LAYA_PRECISION"] ?? "e8") diff --git a/Sources/Decision2048BenchDemo/README.md b/Sources/Decision2048BenchDemo/README.md index 98a3ff7..01750c0 100644 --- a/Sources/Decision2048BenchDemo/README.md +++ b/Sources/Decision2048BenchDemo/README.md @@ -6,11 +6,12 @@ expectimax shortlist to each model. GLiClass compares the candidates in one clas its established `noul` question to each candidate, requiring two passes per move. ```bash -GLICLASS_MODEL_DIR=/path/to/gliclass-assets \ -LAYA_MODEL_DIR="$HOME/Library/Application Support/FluidUse/Models/laya-coreml" \ swift run -c release Decision2048BenchDemo ``` +Both models download their published Core ML assets on first use. To use existing assets, set +`GLICLASS_MODEL_DIR` and/or `LAYA_MODEL_DIR` to their respective model directories. + Set `GAME2048_AUTOLOAD=1 GAME2048_AUTORUN=1` to launch immediately, `GAME2048_SEED=` to select the deterministic starting seed, and `GAME2048_DELAY_MS=` to control visual pacing. Model inference runs alternately so the models do not contend for the Neural Engine; the visual delay is excluded from latency. diff --git a/Sources/FluidUse/GLiClass/GLiClassManager.swift b/Sources/FluidUse/GLiClass/GLiClassManager.swift index af4435c..1137778 100644 --- a/Sources/FluidUse/GLiClass/GLiClassManager.swift +++ b/Sources/FluidUse/GLiClass/GLiClassManager.swift @@ -75,6 +75,19 @@ public actor GLiClassManager { self.tokenizer = tokenizer } + /// Download requested buckets from the Hub into the FluidUse model cache and load them. + /// - Parameter cacheDirectory: The parent Models directory, not the repository subdirectory. + public static func load( + cacheDirectory: URL? = nil, + configuration: Configuration = Configuration(), + progress: GLiClassModelStore.Progress? = nil + ) async throws -> GLiClassManager { + let directory = try await GLiClassModelStore.ensure( + lengths: configuration.lengths, precision: configuration.precision, + cacheDirectory: cacheDirectory, progress: progress) + return try await load(from: directory, configuration: configuration) + } + /// Load `.mlmodelc`/`.mlpackage` buckets and `tokenizer.json` from a local directory. public static func load( from directory: URL, configuration: Configuration = Configuration() diff --git a/Sources/FluidUse/GLiClass/GLiClassModelStore.swift b/Sources/FluidUse/GLiClass/GLiClassModelStore.swift new file mode 100644 index 0000000..9f3ea65 --- /dev/null +++ b/Sources/FluidUse/GLiClass/GLiClassModelStore.swift @@ -0,0 +1,154 @@ +import CryptoKit +import Foundation + +/// Downloads the GLiClass Core ML packages described by the Hub repository's `config.json`. +public enum GLiClassModelStore { + public static let repository = "FluidInference/gliclass-edge-apps-coreml" + public typealias Progress = @Sendable (_ file: String, _ bytes: Int64) -> Void + + static let packageMembers = [ + "Manifest.json", "Data/com.apple.CoreML/model.mlmodel", "Data/com.apple.CoreML/weights/weight.bin", + ] + + struct RepositoryConfig: Decodable { + struct Bucket: Decodable { + let length: Int + let fp16: String? + let lut8: String? + } + + let format: String + let maxOptions: Int + let tokenizer: String + let buckets: [Bucket] + + enum CodingKeys: String, CodingKey { + case format, tokenizer, buckets + case maxOptions = "max_options" + } + + func package(length: Int, precision: String) throws -> String { + guard format == "coreml", maxOptions == GLiClassManager.maximumOptions, + tokenizer == "tokenizer.json" + else { throw GLiClassError.invalidAsset("Unexpected GLiClass repository config") } + guard let bucket = buckets.first(where: { $0.length == length }) else { + throw GLiClassError.invalidAsset( + "GLiClass L\(length) is not published in \(GLiClassModelStore.repository)") + } + let package: String? + switch precision { + case "fp16": package = bucket.fp16 + case "lut8": package = bucket.lut8 + default: package = nil + } + guard let package, + package == (try GLiClassManager.modelName(length: length, precision: precision)) + ".mlpackage" + else { + throw GLiClassError.invalidAsset( + "GLiClass \(precision) L\(length) is not published in \(GLiClassModelStore.repository)") + } + return package + } + } + + /// Ensure selected packages and tokenizer exist under `cacheDirectory/gliclass-edge-apps-coreml`. + /// The Hub config selects the artifacts; published SHA-256 hashes validate cached and downloaded files. + public static func ensure( + lengths: [Int], precision: String = "fp16", cacheDirectory: URL? = nil, progress: Progress? = nil + ) async throws -> URL { + guard !lengths.isEmpty else { throw GLiClassError.invalidAsset("At least one GLiClass bucket is required") } + let root = cacheDirectory ?? LayaModelStore.defaultCacheDirectory() + let directory = root.appendingPathComponent("gliclass-edge-apps-coreml", isDirectory: true) + let configData = try await downloadData("config.json") + let checksumsData = try await downloadData("checksums.json") + let config = try JSONDecoder().decode(RepositoryConfig.self, from: configData) + let checksums = try JSONDecoder().decode([String: String].self, from: checksumsData) + guard let configHash = checksums["config.json"], sha256(configData) == configHash else { + throw GLiClassError.invalidAsset("GLiClass config.json checksum mismatch") + } + var paths = [config.tokenizer] + for length in lengths { + let package = try config.package(length: length, precision: precision) + paths += packageMembers.map { "\(package)/\($0)" } + } + let manager = FileManager.default + try manager.createDirectory(at: directory, withIntermediateDirectories: true) + try configData.write(to: directory.appendingPathComponent("config.json"), options: .atomic) + try checksumsData.write(to: directory.appendingPathComponent("checksums.json"), options: .atomic) + for relative in paths { + guard let expectedHash = checksums[relative], expectedHash.count == 64 else { + throw GLiClassError.invalidAsset("No checksum for \(relative)") + } + let destination = directory.appendingPathComponent(relative) + if manager.fileExists(atPath: destination.path), try sha256(file: destination) == expectedHash { + continue + } + try manager.createDirectory(at: destination.deletingLastPathComponent(), withIntermediateDirectories: true) + progress?(relative, 0) + let temporary = try await downloadFile(relative) + defer { try? manager.removeItem(at: temporary) } + guard try sha256(file: temporary) == expectedHash else { + throw GLiClassError.invalidAsset("GLiClass checksum mismatch for \(relative)") + } + let size = (try manager.attributesOfItem(atPath: temporary.path)[.size] as? NSNumber)?.int64Value ?? 0 + try LayaModelStore.installDownloadedFile(temporary, at: destination) + progress?(relative, size) + } + return directory + } + + private static func url(for relative: String) throws -> URL { + let encoded = relative.addingPercentEncoding(withAllowedCharacters: .urlPathAllowed) ?? relative + guard let url = URL(string: "https://huggingface.co/\(repository)/resolve/main/\(encoded)") else { + throw GLiClassError.invalidAsset("Bad GLiClass download URL for \(relative)") + } + return url + } + + private static func downloadData(_ relative: String) async throws -> Data { + let (data, response) = try await URLSession.shared.data(from: url(for: relative)) + try validate(response, file: relative, size: Int64(data.count)) + return data + } + + private static func downloadFile(_ relative: String) async throws -> URL { + let (temporary, response) = try await URLSession.shared.download(from: url(for: relative)) + do { + let size = + (try FileManager.default.attributesOfItem(atPath: temporary.path)[.size] as? NSNumber)? + .int64Value ?? 0 + try validate(response, file: relative, size: size) + return temporary + } catch { + try? FileManager.default.removeItem(at: temporary) + throw error + } + } + + private static func validate(_ response: URLResponse, file: String, size: Int64) throws { + guard let http = response as? HTTPURLResponse, (200..<300).contains(http.statusCode) else { + throw GLiClassError.invalidAsset( + "Download of \(file) failed (\((response as? HTTPURLResponse)?.statusCode ?? -1))") + } + guard !((http.value(forHTTPHeaderField: "Content-Type") ?? "").contains("text/html")) else { + throw GLiClassError.invalidAsset("Download of \(file) returned HTML") + } + guard size > 0, http.expectedContentLength <= 0 || size == http.expectedContentLength else { + throw GLiClassError.invalidAsset("Download of \(file) has an unexpected size") + } + } + + static func sha256(_ data: Data) -> String { + SHA256.hash(data: data).map { String(format: "%02x", $0) }.joined() + } + + private static func sha256(file: URL) throws -> String { + let handle = try FileHandle(forReadingFrom: file) + defer { try? handle.close() } + var digest = SHA256() + while let chunk = try handle.read(upToCount: 1_048_576), !chunk.isEmpty { + digest.update(data: chunk) + } + return digest.finalize().map { String(format: "%02x", $0) }.joined() + } +} diff --git a/Sources/FluidUseLaya/Game2048Command.swift b/Sources/FluidUseLaya/Game2048Command.swift index 6ddcd3a..057c5a4 100644 --- a/Sources/FluidUseLaya/Game2048Command.swift +++ b/Sources/FluidUseLaya/Game2048Command.swift @@ -76,12 +76,13 @@ struct Game2048Command { let options = try parse(arguments) let manager: GLiClassManager? if options.policy == "gliclass" { - guard let modelDirectory = options.modelDirectory else { - throw GLiClassError.invalidAsset("--policy gliclass requires --model-dir") + let configuration = GLiClassManager.Configuration(lengths: [128], precision: options.precision) + if let modelDirectory = options.modelDirectory { + manager = try await GLiClassManager.load( + from: URL(fileURLWithPath: modelDirectory), configuration: configuration) + } else { + manager = try await GLiClassManager.load(configuration: configuration) } - manager = try await GLiClassManager.load( - from: URL(fileURLWithPath: modelDirectory), - configuration: .init(lengths: [128], precision: options.precision)) } else { manager = nil } diff --git a/Sources/FluidUseLaya/LayaTetrisCommand.swift b/Sources/FluidUseLaya/LayaTetrisCommand.swift index 59c54bb..da62418 100644 --- a/Sources/FluidUseLaya/LayaTetrisCommand.swift +++ b/Sources/FluidUseLaya/LayaTetrisCommand.swift @@ -123,12 +123,13 @@ struct LayaTetrisCommand { layaManager = try await LayaManager.load(configuration: configuration) } } else if options.policy == "gliclass" { - guard let directory = options.modelDirectory else { - throw GLiClassError.invalidAsset("--policy gliclass currently requires --model-dir") + let configuration = GLiClassManager.Configuration(lengths: options.lengths, precision: options.precision) + if let directory = options.modelDirectory { + gliClassManager = try await GLiClassManager.load( + from: URL(fileURLWithPath: directory), configuration: configuration) + } else { + gliClassManager = try await GLiClassManager.load(configuration: configuration) } - gliClassManager = try await GLiClassManager.load( - from: URL(fileURLWithPath: directory), - configuration: .init(lengths: options.lengths, precision: options.precision)) } let gliClassLabels = [ "a poor Tetris placement that creates holes or a dangerous tall stack", @@ -369,8 +370,9 @@ struct LayaTetrisCommand { Plays headless 10x20 Tetris. With --policy laya (default) every legal landing is described in one sentence and scored by laya's P(clean); the best-scoring landing is played. - With --policy gliclass, --model-dir must hold tokenizer.json and the GLiClass Edge Apps v2 - Core ML bucket. The same candidates and descriptions are scored for an apples-to-apples game. + With --policy gliclass, the published GLiClass bucket downloads on first use. --model-dir + can instead point to local tokenizer.json and Core ML bucket assets. The same candidates + and descriptions are scored for an apples-to-apples game. --gliclass-choice compare candidate descriptions in one encoder pass --gliclass-candidates N heuristic prefilter width for choice mode (default 2; max 25) --gliclass-margin P minimum probability margin before GLiClass overrides the heuristic leader diff --git a/Sources/GLiClass2048Demo/Game2048Model.swift b/Sources/GLiClass2048Demo/Game2048Model.swift index 8ee6de9..49db934 100644 --- a/Sources/GLiClass2048Demo/Game2048Model.swift +++ b/Sources/GLiClass2048Demo/Game2048Model.swift @@ -92,13 +92,17 @@ final class Game2048Model: ObservableObject { loadStatus = "Loading GLiClass Edge Apps v2…" Task { do { - guard let directory = ProcessInfo.processInfo.environment["GLICLASS_MODEL_DIR"], !directory.isEmpty - else { throw GLiClassError.invalidAsset("Set GLICLASS_MODEL_DIR for the 2048 demo") } - let precision = ProcessInfo.processInfo.environment["GLICLASS_PRECISION"] ?? "lut8" + let environment = ProcessInfo.processInfo.environment + let precision = environment["GLICLASS_PRECISION"] ?? "lut8" let started = Date() - let loaded = try await GLiClassManager.load( - from: URL(fileURLWithPath: directory), - configuration: .init(lengths: [128], precision: precision)) + let configuration = GLiClassManager.Configuration(lengths: [128], precision: precision) + let loaded: GLiClassManager + if let directory = environment["GLICLASS_MODEL_DIR"], !directory.isEmpty { + loaded = try await GLiClassManager.load( + from: URL(fileURLWithPath: directory), configuration: configuration) + } else { + loaded = try await GLiClassManager.load(configuration: configuration) + } _ = try await loaded.classify( text: "Build the largest tile without filling the board.", labels: ["swipe left: 8 empty cells", "swipe right: 5 empty cells"], diff --git a/Sources/GLiClass2048Demo/README.md b/Sources/GLiClass2048Demo/README.md index 37234d3..8d86a28 100644 --- a/Sources/GLiClass2048Demo/README.md +++ b/Sources/GLiClass2048Demo/README.md @@ -5,9 +5,14 @@ heuristic shortlists the strongest legal swipes; GLiClass compares their resulti in one L128 encoder pass. ```bash -GLICLASS_MODEL_DIR=/path/to/assets GLICLASS_PRECISION=lut8 swift run -c release GLiClass2048Demo +swift run -c release GLiClass2048Demo ``` +The default L128 LUT8 package and tokenizer download from +[FluidInference/gliclass-edge-apps-coreml](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) +on first use and are cached locally. Set `GLICLASS_MODEL_DIR=/path/to/assets` to use local assets, +or `GLICLASS_PRECISION=fp16` for the published FP16 bucket. + Set `GAME2048_AUTORUN=1` to load and play immediately, `GAME2048_SEED=` to choose the deterministic tile sequence, `GAME2048_CANDIDATES=2|3|4` to choose the comparison width, and `GAME2048_MARGIN=0...1` to require a confidence margin before GLiClass overrides the heuristic leader. diff --git a/Sources/LayaTetrisDemo/GameModel.swift b/Sources/LayaTetrisDemo/GameModel.swift index a2d1615..c4fceb6 100644 --- a/Sources/LayaTetrisDemo/GameModel.swift +++ b/Sources/LayaTetrisDemo/GameModel.swift @@ -163,13 +163,16 @@ final class GameModel: ObservableObject { let started = Date() switch policy { case .gliclass: - guard let directory = ProcessInfo.processInfo.environment["GLICLASS_MODEL_DIR"], - !directory.isEmpty - else { throw GLiClassError.invalidAsset("Set GLICLASS_MODEL_DIR for the GLiClass demo") } - let precision = ProcessInfo.processInfo.environment["GLICLASS_PRECISION"] ?? "fp16" - let loaded = try await GLiClassManager.load( - from: URL(fileURLWithPath: directory), - configuration: .init(lengths: [128], precision: precision)) + let environment = ProcessInfo.processInfo.environment + let precision = environment["GLICLASS_PRECISION"] ?? "fp16" + let configuration = GLiClassManager.Configuration(lengths: [128], precision: precision) + let loaded: GLiClassManager + if let directory = environment["GLICLASS_MODEL_DIR"], !directory.isEmpty { + loaded = try await GLiClassManager.load( + from: URL(fileURLWithPath: directory), configuration: configuration) + } else { + loaded = try await GLiClassManager.load(configuration: configuration) + } _ = try await loaded.classify( text: "The piece buries nothing and keeps the stack low.", labels: ["a poor Tetris placement", "a clean Tetris placement"], diff --git a/Sources/LayaTetrisDemo/README.md b/Sources/LayaTetrisDemo/README.md index 9d1891d..897eae8 100644 --- a/Sources/LayaTetrisDemo/README.md +++ b/Sources/LayaTetrisDemo/README.md @@ -9,9 +9,10 @@ milliseconds per move, and calls. swift run -c release LayaTetrisDemo ``` -No Xcode project is needed. Set `GLICLASS_MODEL_DIR` to a directory holding `tokenizer.json` and -`gliclass_edge_apps_fp16_L128_options25.mlpackage`, or select laya and let it download from -`FluidInference/laya-coreml` (`LAYA_MODEL_DIR` skips that download). **Play** scores flat out with +No Xcode project is needed. GLiClass downloads its L128 FP16 package and tokenizer from +`FluidInference/gliclass-edge-apps-coreml` on first use. Set `GLICLASS_MODEL_DIR` to a local asset +directory to skip that download. Laya downloads from `FluidInference/laya-coreml` unless +`LAYA_MODEL_DIR` selects a local directory. **Play** scores flat out with Core ML on CPU + ANE and continues on the next seed after a top-out. SwiftUI presents the evolving board at the display refresh rate while the model stays hot. The *pause per piece* slider can expose individual moves, but long pauses let the Neural Engine idle and raise measured latency. The *delay per scored @@ -19,8 +20,8 @@ landing* slider slows laya's candidate-by-candidate scoring so it can be watched outline); the chosen landing is drawn in green. The GLiClass training, Core ML conversion, compression, and parity reports are in -[mobius PR #101](https://github.com/FluidInference/mobius/pull/101). The GLiClass packages are not yet -auto-downloaded, so build or obtain those assets before running this policy. +[mobius PR #101](https://github.com/FluidInference/mobius/pull/101). The published GLiClass packages +and runtime config are at [Hugging Face](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml). **Policy** switches among GLiClass, laya, a feature-weighted heuristic, and random play. The harness toggle applies to every policy. See [Benchmarks.md](../../Benchmarks.md) for the reported runs. diff --git a/Tests/FluidUseTests/Fixtures/gliclass-hub-config.json b/Tests/FluidUseTests/Fixtures/gliclass-hub-config.json new file mode 100644 index 0000000..7183b3b --- /dev/null +++ b/Tests/FluidUseTests/Fixtures/gliclass-hub-config.json @@ -0,0 +1,46 @@ +{ + "format": "coreml", + "library_name": "coreml", + "pipeline": "dynamic-label-classification", + "source": { + "repo": "knowledgator/gliclass-edge-v3.0", + "revision": "df03993a2ed98e5e4a0d2dd7efbbd105abe874cf", + "license": "apache-2.0" + }, + "variant": "FluidUse application-tuned v2", + "parameters": 32705154, + "max_options": 25, + "sequence_format": "[CLS] (<