diff --git a/Documentation/GameDemos.md b/Documentation/GameDemos.md new file mode 100644 index 0000000..3e46c8a --- /dev/null +++ b/Documentation/GameDemos.md @@ -0,0 +1,339 @@ +# Decision game demos + +Ideas for comparing small decision models in observable, repeatable environments. +These are proposed demos, not games currently shipped in FluidUse. Tetris, 2048, +and Snake are the starting examples from the discussion. Flappy Bird has been +tried ([results](#flappy-bird-trial-results)): no text-state model played it +usefully, so real-time control games are [ruled out](#ruled-out-real-time-games). +Connect Four ([results](#connect-four-trial-results)) and Snake +([results](#snake-trial-results)) have also been tried: both +confirm that these models fail where one wrong move loses or where play needs +lookahead. Minesweeper is kept as a [benchmark mode](#minesweeper-benchmark-mode) +rather than a live demo. +The [model inventory](Models.md) identifies possible sub-1B models +and distinguishes text-state models from vision models. + +## Recommended build order + +This is a judgment about **demo value for FluidUse**, now informed by six +trials (Tetris, 2048, Flappy Bird, lane runner, Connect Four, Snake). The +pattern across them: these models do well when each move is a short-horizon +choice among a few described options and a mediocre move costs points rather +than the game (Tetris with a heuristic shortlist, 2048). They fail when one wrong +move is fatal or when a good move needs lookahead (Flappy Bird, lane runner, +Connect Four, Snake). The ranking therefore favors forgiving, +turn-based puzzles with a visible score. + +| Rank | Game | Main reason to demo it | +| ---: | --- | --- | +| 1 | Drop-merge ("2048 blocks") | One of five columns per drop, short horizon, forgiving; closest to what already works. | +| 2 | Threes! | 2048's predecessor; reuses most of the `Game2048` engine. | +| 3 | Codenames (guesser) | Semantic association, the closest match to how these models are trained; no lookahead. | +| 4 | Wordle-style word game | Compact text state and a fixed candidate list; a bad guess costs a turn, not the game. | +| 5 | Block Blast / 1010! | Placement puzzle; the Tetris shortlist pattern applies directly. | +| 6 | Suika (watermelon) | Drop-slot choice with physics; very watchable, but scores are noisy. | +| 7 | Minesweeper (benchmark mode) | Exact mine probabilities give a calibration benchmark; not a live demo (one wrong click ends it). | +| 8 | MiniWoB-style web tasks | Closest to FluidUse's computer-use purpose, with objective task success. | +| 9 | Chess (position challenge) | Static ranked-move scoring works; full games would repeat the Connect Four result. | +| 10 | Battleship | Hidden-information search, but slower to watch. | +| 11 | Sokoban / Boxoban | Needs multi-step lookahead; expect the Connect Four and Snake pattern. | +| 12 | MiniGrid DoorKey | Multi-step planning. | +| 13 | Hanabi | Cooperative partial information, hard to explain at a glance. | +| 14 | Tower defense | Large action and state space. | +| 15 | FrozenLake | Useful stochastic baseline, visually less compelling. | +| — | Tetris, 2048 | Already built. | +| — | Connect Four, Snake, multiplayer Snake arena | Tried or ruled out; see the trial sections. | + +## Demo list + +| Status | Game | Model's decision | What it reveals | Suggested state | +| --- | --- | --- | --- | --- | +| Candidate | **Sokoban / Boxoban** | Pick a legal move or push. | Planning and irreversible traps. | Grid, player, crates, goals, and legal moves. | +| Candidate | **Chess** | Choose or rank legal moves. | Tactical judgment, position evaluation, and planning across turns. | FEN, side to move, legal moves, and remaining time; a board image only for vision models. | +| Candidate (benchmark) | **Minesweeper** | Say whether a frontier cell is safe. | Calibration against exact mine probabilities. | 3×3 or 5×5 neighborhood per cell; see [benchmark mode](#minesweeper-benchmark-mode). | +| Candidate | **Wordle-style word game** | Choose the next valid guess. | Information gathering versus an immediate attempt to solve. | Previous guesses, color feedback, and a fixed candidate-word list. | +| Tried | **Connect Four** | Choose a non-full column. | Short tactical lookahead. Every model lost 0–20 to a rule-based heuristic ([results](#connect-four-trial-results)). | Compact board, player to move, and legal columns, optionally labeled with tactics. | +| Candidate | **Drop-merge** | Choose the column for the next tile. | Short-horizon merging with a clear score. | Small numeric grid, next tile, and each column's resulting merges. | +| Candidate | **Threes!** | Choose a legal slide. | Like 2048, with harder merge rules. | Board, next tile, and each legal resulting board. | +| Candidate | **Suika (watermelon)** | Choose a drop slot. | Merging under physics; visual. | Fruit positions summarized per slot, next fruit. | +| Candidate | **Block Blast / 1010!** | Choose a placement from a shortlist. | Board management; same pattern as Tetris. | Grid, pieces in hand, and shortlisted placements with line clears. | +| Candidate (benchmark) | **Blackjack** | Hit, stand, or double. | Throughput and win rate against basic strategy. | Hand, dealer upcard, and legal actions. | +| Ruled out | **Multiplayer Snake arena** | Choose direction while several snakes move simultaneously. | Would compound the single-player Snake failure ([assessment](#snake-trial-results)). | Board, all visible snakes, food, and legal directions. | +| Candidate | **Codenames** | Choose a clue or a guess from a fixed set. | Semantic association under constraints. | Visible words, team, prior clues, and legal choices. | +| Candidate | **Hanabi** | Play, discard, or give a legal hint. | Cooperation with incomplete information. | Only what the acting player may observe. | +| Candidate | **MiniGrid DoorKey** | Turn, move, pick up, or open. | Multi-step planning when a key must be found before reaching the goal. | Partial grid observation, carried item, and door state. | +| Candidate | **FrozenLake** | Choose one of four directions. | Planning under uncertain movement when the lake is slippery. | Map, position, goal, holes, and known movement rules. | +| Candidate | **Battleship** | Choose an untried target square. | Search under hidden information and using feedback from earlier turns. | Hits, misses, sunk ships, and remaining legal squares. | +| Candidate | **Tower defense** | Place, upgrade, sell, or wait. | Resource allocation with delayed effects. | Map, waves, towers, budget, and legal placements. | +| Candidate | **MiniWoB-style web tasks** | Choose an element and operation. | Practical computer use with a clear success condition. | Accessibility element table and task goal; screenshots for vision models. | +| Existing example | **Tetris** | Choose a placement or movement. | Long-term board management. | Board, current piece, next piece if allowed, and legal placements. | +| Existing example | **2048** | Choose a legal slide. | Repeated choices with random future tiles. | Board and each legal resulting board before the random spawn. | +| Tried | **Snake** | Choose direction. | Path planning while avoiding self-traps; best model (Lex) ate 112 food over 10 games versus 305 for the heuristic ([results](#snake-trial-results)). | Grid, body, food, and current direction. | + +The merge puzzles and Codenames play to what these models showed in Tetris and +2048: choosing well among a few described options without lookahead. Chess +positions and Minesweeper stay useful as static benchmarks. Connect Four, Snake, +and Sokoban test lookahead, where the trials so far show these models fail. + +Chess has two useful modes. A **position challenge** gives every model the same +FEN and legal move list, then compares its ranked moves with published reference +values. The Decision Index already includes this kind of static test as +[ChessBench](https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/main/data/index.json). +A **full-game demo** pairs models against each other at the same clock setting +and matched starting positions. Record legal-move rate, move time, game result, +and engine evaluation loss per move. Supply legal moves from a chess rules engine +so the model is judged on choosing among them, not on formatting notation. + +## Competitive games + +Make **model vs model** the main viewer mode. The following modes use the same +decision interface, with fixed-opponent runs retained as a diagnostic: + +| Mode | Purpose | How it works | +| --- | --- | --- | +| Model vs model | Main spectator match. | Run a round robin with paired colors and matched opening positions. Show wins, draws, losses, illegal or timed-out decisions, and move time. | +| Same model vs itself | Self-play showcase. | Launch two independent instances of the same checkpoint. Swap colors across paired games and vary openings. If choices are sampled, give each instance an independent random seed and show the sampling setting. | +| Multi-model arena | Actual battle royale. | Put three or more models in one simultaneous-action game, such as multiplayer Snake. Give every model the same board snapshot each tick, collect actions independently, then resolve them together at the deadline. Rotate spawn positions across matches. | +| Human vs model | Interactive side mode. | The person picks a side and a legal move; show the model's top choices, probabilities when available, and response time. Do not count these self-selected games in model rankings. | +| Model vs fixed opponent | Diagnostic. | Each candidate faces the same version and settings of a reference policy. Run paired games from the same starting positions with colors or first turn swapped. | + +For competitive demos, note the [Connect Four result](#connect-four-trial-results): +every model lost to a simple heuristic, so model-vs-model matches would pit weak +players against each other. If a duel is still wanted, Connect Four is the +cheapest to run (the headless check already exists); multiplayer Snake is ruled +out, and chess is better used as a static position challenge. +An existing [MIT-licensed multiplayer Snake game](https://github.com/simondiep/node-multiplayer-snake) +has spectator mode, bots, and adjustable speed; evaluate it as a reusable base +instead of expanding PlayJev's single-player Snake from scratch. Connect Four +has a small action set and short games. Use a rules engine to provide legal +columns and a fixed search policy only for diagnostics. [chess.js](https://github.com/jhlywa/chess.js) +can provide legal moves, validation, and game-end detection in a browser demo; +[Stockfish](https://github.com/official-stockfish/Stockfish) can be the fixed +reference opponent for diagnostics at a stated strength and time budget. Models see the same FEN, +move history limit, legal move list, and clock information. Rotate colors and +use a fixed set of opening positions. Keep any engine evaluation out of the +model's input; calculate it afterward for the viewer. + +In self-play, identical deterministic policies can repeat the same line or draw +often. Vary the opening positions and show that behavior honestly; optional +sampling makes matches more varied, but its temperature must be declared and +held fixed across models. For rankings, use many paired matches and record the +model version, side or spawn, opening or map seed, action deadline, and full +move log. A single spectacular match is a demo clip, not a performance estimate. + +For hidden-information games, keep roles and observations honest. In Battleship, +each player sees only its own ships and prior shots. In Codenames, score clue +giver and guesser separately, with a fixed partner for comparable runs. Hanabi +is cooperative rather than adversarial: use fixed teammate policies and report +team score, not win rate against another model. Record opponent version, seeds, +starting positions, rules, action deadline, and every move so a result can be +replayed. + +Wordle and Battleship add information-gathering decisions; Connect Four is a +compact turn-based duel. For reusable planning +environments, [MiniGrid DoorKey](https://minigrid.farama.org/environments/minigrid/DoorKeyEnv/) +and [Gymnasium FrozenLake](https://gymnasium.farama.org/main/environments/toy_text/frozen_lake/) +provide configurable tasks with small discrete action spaces. Start each model +from the same puzzle or seeded map, and keep its observations limited to what +the game rules permit. + +## Reuse existing games + +The goal is to adapt playable games, not rebuild their physics, visuals, and +controls. A demo adapter should expose `reset(seed)`, `observe()`, `legalActions()`, +`step(action)`, `score()`, and `done()`. It can call an existing game's input +functions or simulate its keyboard controls. Keep gameplay and assets upstream +where possible; record the upstream revision and changes to the adapter. + +| Source | Good for | Reuse notes | +| --- | --- | --- | +| [PlayJev game harness](https://github.com/OmniJev/PlayJev) | Sokoban, Tetris, Snake, and 2048; its other six games are real-time and ruled out. | Already exposes seeded `start`, `step`, `frame`, `score`, `done`, and actions. Each vendored game has its own license; check art and levels separately before publishing. | +| [Node Multiplayer Snake](https://github.com/simondiep/node-multiplayer-snake) | Multi-model survival arena. | MIT-licensed browser game with spectator mode, bots, and adjustable speed. Adapt its player controllers to model actions and add seeded resets. | +| [MiniGrid](https://minigrid.farama.org/environments/minigrid/) and [Gymnasium](https://gymnasium.farama.org/main/environments/) | DoorKey, FrozenLake, and other compact decision tasks. | Existing reset/step environments; add a viewer and a model input adapter. | + +### Start with PlayJev + +Use [OmniJev/PlayJev](https://github.com/OmniJev/PlayJev) as the harness for +Sokoban and the existing examples, rather than implementing them again. Its browser +hook provides `window.pj.start(seed)`, `step(action)`, `frame()`, `score()`, +`done()`, and an action list. The repository also includes random and teacher +policies, recorded runs, and a [0.8B vision model](https://huggingface.co/OmniJev/PlayJev-0.8B) +that can serve as a visual reference. Keep the game, seed, tick rate, and action +list fixed while swapping the decision provider. For text-only models, add a +state extractor for each game and report text-state and pixel-input results in +separate tracks. Re-run all compared models in the same harness; PlayJev's +published scores are background context, not directly comparable results. + +Before vendoring or publishing a game, review its own license and bundled art. +PlayJev documents the licenses of its ten vendored games and notes that some +sprites are owned by third parties. + +## Flappy Bird comparison specification + +1. The game engine advances at a fixed rate. Models receive a state at the same + decision interval and choose `FLAP` or `COAST`. The engine owns physics and + collision rules. +2. Run each model separately on the same seeded pipe sequences. Fix game speed, + decision interval, state schema, action semantics, and deadline. Give vision + models the same rendered frame; give text models the same structured facts. + Report vision and text-state results as different input tracks. +3. Show each action and its probability when available, plus the time it took. + An action probability expresses the model's preference between `FLAP` and + `COAST`; it is not a survival probability. +4. Keep the clock running during inference. If a result misses the deadline, + apply a declared fallback action and count a missed deadline. +5. Report pipes passed, survival ticks, median and p95 decision latency, missed + deadlines, and number of calls for every seed. Add a random policy and a + simple physics controller as reference runs. Publish the seed list and + per-run results before reporting aggregates. + +The [Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) +measures a frozen panel of static decision tasks. Its current score excludes +interactive environments, including Boxoban, Hanabi, Codenames, and MiniWoB++. +Use matched-seed game results to establish game performance; Index rank is only +a way to find candidate models. [PlayJev](https://github.com/OmniJev/PlayJev) +is a separate 0.8B vision model with published results on several of these games, +including Flappy Bird. Its published numbers use its own harness and should not +be placed on the same chart as new runs until the harness matches. + +## Flappy Bird trial results + +Flappy Bird was built and run as +[FluidUse #10](https://github.com/FluidInference/FluidUse/pull/10) (`GLiClassFlappyDemo` app and +`FlappyBirdCheck` headless runner). Each model got a text state plus a 300 ms +physics forecast for each action, and chose `FLAP` or `COAST` every six frames (10 Hz). +Without the safety guard, on seeds 1–4 (Apple M5 Pro): + +| Policy | Pipes passed | Combined survival | +| --- | ---: | ---: | +| GLiNER 2.5 multilingual W8 | 1 | 10.98 s | +| Kev 0.6B, Decision 1.0 Lex | 0 | 9.03 s each | +| Kev 0.5B, Decision 1.0 Kai, GLiNER 2.5 small | 0 | 8.5–8.7 s | +| LFM2.5-350M-RLCD, Jeff | 0 | 6.7–7.1 s | +| Laya, GLiClass, NanoJev, Verdict, GLiNER 2.5 base | 0 | 5.4–5.9 s | +| Physics heuristic | 40 | 80 s (four 20 s caps) | + +None of the 13 models is a usable controller. Most ignored the forecast. GLiClass, +for example, flapped on almost every call, and LFM always chose the option listed second. +The safety guard reached the cap but overrode 106–131 of 200 choices, so that +result measures the guard. LFM (~150 ms) and NanoJev (~78 ms) also miss the +100 ms decision period. Per-tick timing control does not suit +label-choosing models. A fairer follow-up would ask a yes/no safety question or +query only at hard decision points. + +## Ruled out: real-time games + +The Flappy Bird result applies to every game that needs control on each frame +or tick. These games have more actions than Flappy Bird, need inputs held +for exact frame counts, or have many fatal mistakes per run: + +| Game | Why it is ruled out | +| --- | --- | +| Mario-style platformer | Left/right/jump/run combinations; jump height depends on hold duration; many fatal mistakes per level. | +| Subway Surfers-style runner | Lane, jump, and slide timing against obstacles at speed. | +| Frogger-style crossing | Timing windows around moving hazards. | +| Pac-Man | Per-tick direction under chasing ghosts. | +| Space Invaders | Continuous aiming and dodging. | +| Breakout / Pong | Paddle control against ball speed. | +| Racer | Per-frame steering corrections. | + +A "decision point" variant could reuse some of them. It pauses at each obstacle and +offers consequence-labeled options while scripted movement handles the rest. In +that setup the game logic does most of the work, so it would say little about +the models. + +## Connect Four trial results + +A headless `ConnectFourCheck` (branch `feat/connect4-model-comparison`, stacked +on [FluidUse #11](https://github.com/FluidInference/FluidUse/pull/11); not yet a +PR) played each model against a rule-based heuristic: win, then block, then +never hand over a win, then a window score, then the center. Each of 10 seeds +gave two games, with the model moving first and then second, after two seeded +random opening moves. Legal columns were rotated each move. In **describe** mode +each column was labeled with its tactics ("wins now", "blocks their win", "gives +them a win", "makes three"); in **raw** mode only "column N" was shown. The +board used a compact 6-line text grid to fit 128-token models. Kai and Lex +(3 slots) and NanoJev (4) saw the heuristic's top columns. Apple M5 Pro: + +| Model | Wins (of 20, describe / raw) | ms per move (median) | Missed blocks (describe / raw) | +| --- | ---: | ---: | --- | +| GLiClass LUT8 | 0 / 0 | 2.0 | 17 / 18 | +| Laya E8 | 0 / 0 | 5.0 | 15 / 20 | +| Decision 1.0 Kai (top-3 shortlist) | 0 / 2 | 5.3 | 17 / 14 | +| Decision 1.0 Lex (top-3 shortlist) | 0 / 0 | 5.4 | 15 / 17 | +| GLiNER 2.5 small / base | 0 / 0 | 6.5 / 8.8 | 18–20 | +| Verdict FP16 | 0 / 0 | 10.8 | 20 / 17 | +| Jeff FP16 | 0 / 0 | 12–17 | 19 / 19 | +| Kev 0.5B / 0.6B | 0 / 0 | 16–19 | 1–14 / 11–14 | +| NanoJev (top-4 shortlist) | 0 / 0 | 75 | 19 / 20 | +| LFM2.5-350M-RLCD | 0 / 0 | 108–116 | 14 / 16 | +| Heuristic vs itself | 9 wins, 2 draws | — | 0 | +| Random | 0 | — | 20 | + +A missed block is a move where the opponent could win next turn, a blocking +column was offered, and the model chose another. Labeling the blocking column +barely helped: most models ignored "blocks their win". Kev 0.5B read the labels +(one missed block) but kept choosing "gives them a win" columns (10 blunders). +Without labels, position bias dominated: Kev 0.5B chose the first option on all +89 moves and Kai the third on 97 of 122. GLiNER 2.5 multilingual is excluded: +its tokenizer put 55–79 moves over the 128-token budget. The speed ordering is +clear, but no model plays Connect Four usefully, so it is not a demo. + +## Snake trial results + +A headless `SnakeCheck` (branch `feat/snake-model-comparison`, stacked on the +Connect Four branch; not yet a PR) steered the snake on a 10×10 board over 10 +seeds. Each game ended on a crash, after 300 steps, or after 100 steps without +food. Every step offered the three non-reversing directions, rotated. In +**describe** mode each was labeled with its facts ("crash", "eats food", "closer +to food" / "away from food", "safe" / "dead end" from a flood fill); in **raw** +mode only the direction was shown. The state was one line: length, heading, +food offset, and wall distances. The heuristic survives, avoids dead ends, then +chases food. Apple M5 Pro: + +| Model | Food (10 games, describe / raw) | Mean steps (describe) | Crashes (avoidable), describe | ms per move (median) | +| --- | ---: | ---: | --- | ---: | +| Heuristic | 305 | 297.6 | 1 (0) | — | +| Decision 1.0 Lex | **112** / 4 | 126.7 | 10 (3) | 4.1 | +| GLiClass LUT8 | 100 / 0 | 74.6 | 10 (7) | 1.5 | +| Jeff FP16 | 91 / 2 | 207.5 | 3 (2) | 8.6 | +| Decision 1.0 Kai | 72 / 2 | 52.4 | 10 (10) | 4.2 | +| Verdict FP16 | 53 / 8 | 37.1 | 10 (10) | 7.9 | +| NanoJev | 21 / 5 | 34.6 | 9 (9) | 58.4 | +| GLiNER 2.5 multilingual | 14 / 7 | 17.7 | 10 (10) | 6.8 | +| Kev 0.6B / 0.5B | 10 / 4, 5 / 2 | 118.2 / 6.8 | 0, 10 (10) | 11.5 / 9.3 | +| GLiNER 2.5 base / small | 2 / 6, 0 / 8 | 100.2 / 92.2 | 0, 1 (1) | 9.2 / 5.7 | +| Laya E8 | 2 / 2 | 4 | 10 (10) | 3.7 | +| LFM2.5-350M-RLCD | 2 / 5 | 4 | 10 (10) | 56.1 | +| Random | 2 | 26.7 | 10 (10) | — | + +An avoidable crash is a crash when a non-crashing move was offered. Lex is the +best model, at 37% of the heuristic's food, and GLiClass is close behind at +under 2 ms per move; they are the only two that both chase food and sometimes +heed "crash". Kai and Verdict close in on the food nearly every time (97–98%) +but took the "crash" option in every game. Jeff survived longest but skipped +safe food one step away 164 times. Kev 0.6B and GLiNER base and small rarely +crash but wander until the no-food limit. Laya, LFM, and Kev 0.5B drive +straight into the wall in about four steps, matching an earlier local check +where Laya ate 2 food to a heuristic's 27. In raw mode every model ate 8 food +or fewer and crashed within about 4–11 steps, sooner than random, so the +describe-mode skill comes from reading the labels, not the board. + +Snake is not recommended as a demo. One mistake ends the game over hundreds of +moves. Even with the flood-fill check done for the model (the "dead end" and +"crash" labels, which alone are essentially the winning strategy), the best model +reached about a third of the heuristic, and most crashes were into moves +labeled "crash". It is also turn-based, so model speed does not show. + +## Minesweeper: benchmark mode + +As a live game, Minesweeper has the same flaw as Snake: one wrong click ends +it, and most correct moves come from exact counting logic that a solver does +better. As a benchmark it is distinctive. For every frontier cell, ask the model +whether the cell is safe from its 3×3 or 5×5 neighborhood, which fits a +128-token budget and is a yes/no question every model supports. A solver gives +the exact mine probability of each cell, so the result measures calibration +(does "80% safe" mean 80%?) as well as accuracy and cells per second. That suits +Verdict and Kev, which claim calibrated probabilities. Show a board view that +highlights each model's picks, but report the sweep over thousands of positions, +not single games. diff --git a/Documentation/Models.md b/Documentation/Models.md new file mode 100644 index 0000000..654f648 --- /dev/null +++ b/Documentation/Models.md @@ -0,0 +1,123 @@ +# Models + +What FluidUse runs today and the open-weight candidates worth converting next. +Sizes are fp16 on disk unless stated otherwise. Traction was checked on +2026-09-21; licenses were read from each model card. The [Decision Index 0.1](https://huggingface.co/spaces/multimodalart/jev-decision-index) +snapshot below was checked on 2026-09-22. Its scores measure a frozen set of static +decision tasks, not game play. + +## Converted and integrated + +| Model | Params / fp16 | License | Role in FluidUse | +| --- | ---: | --- | --- | +| [cua-ai/cua-s1-forms](https://huggingface.co/cua-ai/cua-s1-forms) → [FluidInference/cua-s1-forms-coreml](https://huggingface.co/FluidInference/cua-s1-forms-coreml) | 706K / 1.4 MB | MIT | Matches each form field to a value from the document: `fill`, `check`, `click`, or `skip`. ~1 ms per decision on the Neural Engine. | +| [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) → [FluidInference/laya-coreml](https://huggingface.co/FluidInference/laya-coreml) | 322M / ~644 MB | Apache-2.0 | `LayaManager` answers text-state `choice`, `score`, and `noul` questions. Core ML buckets at 128/256/512/1024 tokens; Tetris and 2048 demos. | +| [knowledgator/gliclass-edge-v3.0](https://huggingface.co/knowledgator/gliclass-edge-v3.0) → [FluidInference/gliclass-edge-apps-coreml](https://huggingface.co/FluidInference/gliclass-edge-apps-coreml) | 32.7M / 65.7 MB (33.0 MB LUT8 L128) | Apache-2.0 | Application-tuned `GLiClassManager` compares 2–25 supplied labels in one pass; Tetris, 2048, and GLiClass-vs-Laya demos. FP16 buckets at 128/256/512 tokens. | + +CUA-S1-FORMS is a one-pass option scorer (a "System One" model in TypeSafe's terms), not an +LLM. It is a byte-level 2-layer Transformer with the option-attention head from +[jevlike](https://github.com/vinnylarouge/jevlike), trained by Cua on 10,000 synthetic forms. +It does not read goals, write text, or reason about dropdown options. Laya and +GLiClass handle more general decisions but the form-filling app still uses CUA-S1-FORMS. +Apple's built-in Foundation Models are not a FluidUse conversion and are not +included in this inventory. + +## Candidates to convert + +Under 1 GB, permissively licensed, untouched so far. Ordered by how much each adds. + +| Model | Params / fp16 | License | Purpose | Evidence | +| --- | ---: | --- | --- | --- | +| [cklxx/laya-browser](https://huggingface.co/cklxx/laya-browser) v10s | 322M / 644 MB | Apache-2.0 | Picks which element to act on and which operation (`CLICK`, `TYPE_TEXT`, `SELECT`, `DONE`) from a goal and the page's element table. The role TypeSafe's Jev plays in browser agents. | Element top-1 0.63 over ~45 candidates, operation accuracy 0.88, 50% on 16 live browser tasks. Same architecture as the laya port above, so it is a weight bucket plus a sequence format, not a new conversion. | +| [convaiinnovations/laya-typed-decisions](https://huggingface.co/convaiinnovations/laya-typed-decisions) | 421M / 843 MB | Apache-2.0 | Dropdown choice, yes/no verification, and ordered-scale questions over a profile. | 0.766 on the typed-decisions benchmark, above Jev 1.13's published 0.727. Loads through `LayaManager`. | +| [urchade/gliner_small-v2.1](https://huggingface.co/urchade/gliner_small-v2.1) | ~150M / ~300 MB | Apache-2.0 | Pulls name, phone, employer, and any other labeled span out of an unstructured résumé or letter, replacing the `Label: value` parser. | 19k downloads a month; GLiNER repo 3.9k stars. | +| [microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) | 231M / 463 MB | MIT | Captions icons and regions in a screenshot, for apps with no accessibility tree. OmniParser's captioner. | 3M downloads a month. Only useful once a screenshot path exists. | + +Fallbacks, also under 1 GB and untouched: + +| Model | Params / fp16 | License | Purpose | +| --- | ---: | --- | --- | +| [urchade/gliner_multi_pii-v1](https://huggingface.co/urchade/gliner_multi_pii-v1) | ~300M / ~580 MB | Apache-2.0 | Same extractor, tuned for identity documents and personal-data fields. | +| [torontodeveloper/mind2web-candidate-ranker](https://huggingface.co/torontodeveloper/mind2web-candidate-ranker) | 184M / 369 MB | MIT | Ranks page elements by how likely each is the next click target; no operation choice. Recall@10 93% on unseen Mind2Web domains. | +| [shreyanbr/system-one-distilled](https://huggingface.co/shreyanbr/system-one-distilled) | 71M / 142 MB | Apache-2.0 | Smallest generic typed-decision model, for routing and gating where laya is overkill. | + +### More S1-class specialists + +Nothing else at CUA-S1-FORMS's scale exists with weights. The way to get more is to train +them: Cua's trainer, synthetic generator, and evaluator ship under MIT in +[trycua/cua `libs/cua-s1`](https://github.com/trycua/cua/tree/main/libs/cua-s1), and +[jevbetter](https://github.com/olanotolu/jevbetter) accepts the same JSONL with a stronger +encoder. Closed-vocabulary tasks that fit the shape, each 1 to 5 MB: + +- Submit-or-advance: which button submits, continues, or dismisses a dialog. +- Field-role tagging: search box, login form, cookie-banner accept. +- Clutter-or-ad per element. + +## Demo ideas + +| Model | Demo | What the viewer sees | +| --- | --- | --- | +| laya-browser | Goal-driven browsing in Safari: "book the cheapest nonstop to New York in October" | Each pick and its probability logged while the agent steps through Google Flights, fully offline. | +| laya-typed-decisions | Dropdowns and consent fields in an HR onboarding flow | "Years of experience", "work authorization", and "I agree" resolved from the profile with a confidence next to each. | +| gliner_small-v2.1 | Drop a real résumé PDF instead of a `Label: value` profile | The entity list fills from prose in under a second, then S1 fills the form from it. | +| gliner_multi_pii-v1 | Photo of an ID or insurance card via Vision OCR | Name, date of birth, member ID pulled from the card into a patient-intake form. Nothing leaves the machine. | +| Florence-2-base | Screenshot of an app with no accessibility tree | Icons and buttons labeled in place, then a decision made on those labels. | +| mind2web-candidate-ranker | "Where would you click next" overlay | Top five elements highlighted with scores, no action taken. | +| system-one-distilled | Routing benchmark | Hundreds of tickets classified per second with latency per call. | + +### Game decisions + +See [Decision game demos](GameDemos.md) for the game list, the Flappy Bird +comparison specification, and the matched-seed evaluation protocol. Flappy Bird +has been tried with every integrated model, including Kev, LFM, and Lex; none played it +usefully ([results](GameDemos.md#flappy-bird-trial-results)). Connect Four and +Snake gave the same answer: every model lost 0–20 to a simple heuristic at +Connect Four, and at Snake the best model (Lex) ate 112 food to the heuristic's +305 over 10 games. The +recommended next demos are forgiving merge puzzles such as drop-merge and Threes; +Minesweeper is kept as a calibration benchmark +([details](GameDemos.md#recommended-build-order)). + +### Promising Index models below 1B parameters + +These are **parameter-count** candidates from the [Index data](https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/main/data/index.json), +not models already converted for FluidUse. The Index score is its balanced static +score out of 100; it is included to identify candidates, not to predict game +performance. License and input type come from the linked model cards. Approximate +fp16 sizes include only weights (2 bytes per parameter), before runtime overhead. + +| Model | Params / fp16 | Index score | Input and best demo role | Status | +| --- | ---: | ---: | --- | --- | +| [Kev-0.6B](https://huggingface.co/jaredpalmer/kev-0.6b) | 596M / ~1.19 GB | 31.30 | Text state, typed `choice` / `score` / `noul`; first generic Flappy Bird decision candidate. | Apache-2.0; LoRA and pointer head require its Qwen3 base and a new runtime port. | +| [LFM2.5-350M-RLCD](https://huggingface.co/notnotsamuel/LFM2.5-350M-RLCD) | 354M / ~709 MB | 25.79 | Text decision baseline for Flappy Bird or routing. | `lfm1.0` custom license: review terms before conversion or redistribution. | +| [GLiNER2.5-base](https://huggingface.co/fastino/gliner2.5-base-v1) | 194M / ~387 MB | 24.70 | Text extraction and classification; best fit for form/document decisions. | Apache-2.0; needs a separate GLiNER runtime, not a direct game controller. | +| [GLiNER2.5-small](https://huggingface.co/fastino/gliner2.5-small-v1) | 74M / ~148 MB | 23.93 | Smaller text extraction/classification baseline. | Apache-2.0; different from the GLiNER small v2.1 candidate above. | +| [GLiNER2.5-multi](https://huggingface.co/fastino/gliner2.5-multi-v1) | 287M / ~575 MB | 22.42 | Multilingual form/document decisions. | Apache-2.0; separate GLiNER runtime. | +| [Decision-1.0-Lex](https://huggingface.co/llm-semantic-router/decision-1.0-lex) | 308M served in Index / ~616 MB | 19.57 | Text choices and scores; operational tasks and a Flappy Bird text-state trial. | Apache-2.0; card describes a 572M full checkpoint, so verify the artifact before sizing a port. | +| [Decision-1.0-Kai](https://huggingface.co/llm-semantic-router/decision-1.0-kai) | 308M served in Index / ~616 MB | 18.37 | General text decision comparator for Lex. | Apache-2.0; card calls it 0.6B, so confirm the exact artifact before conversion. | +| [Laya](https://huggingface.co/convaiinnovations/laya) | 421M / ~843 MB | 16.39 | Text typed decisions; comparison with the existing multilingual Core ML port. | Apache-2.0; the Index tests the English checkpoint, not `laya-multilingual`. | + +The Index also includes [NanoJev](https://huggingface.co/C-Tianyu/NanoJev) +(596M, 26.19), but its Hub card does not declare a license; keep it out of the +conversion shortlist until terms are clear. [Kev-0.5B](https://huggingface.co/jaredpalmer/kev-0.5b) +(494M, 30.34) is explicitly described by its author as a superseded prototype. +Neither is a first-choice port. All Index models in this table take text input; +there is no evidence in the Index that they can read Flappy Bird screenshots. + +[PlayJev-0.8B](https://huggingface.co/OmniJev/PlayJev-0.8B) is a separate, +Apache-2.0 **vision** candidate under 1B parameters, not a scored Index entrant. +It has a published Flappy Bird run and can take rendered frames, making it the +most direct visual comparator. Its fp16 weights are roughly 1.6 GB, above the +1 GB fp16 limit used for the main FluidUse conversion shortlist. Its published +Flappy Bird score must not be compared directly with a new harness run; use the +same physics, seeds, action timing, and scoring for every model in the demo. + +## Ruled out + +- **TypeSafe Jev**: hosted API only, no weights. +- **Screenshot grounders at 2B and up** (GUI-Owl-1.5-2B, UI-TARS-2B, ShowUI-2B, GUI-Actor-2B, Holo2-4B): 4 to 9 GB fp16, no Core ML precedent for their vision towers. +- **Non-commercial licenses**: Holo1.5-3B (Qwen Research), pngwn/system-one-qwen3.5-4b-scorer (CC-BY-NC), laya-vision-smolvlm and laya-grounded (CC-BY-NC). +- **AGPL**: OmniParser `icon_detect`, macpaw-research/yolov11l-ui-elements-detection. +- **No weights**: Apple Ferret-UI Lite, Ferret-UI 2, UI-JEPA. +- **"1 MB" heads that need a 0.6B backbone**: samatv256/mini-Jev and similar. +- **Decoder-based Jev clones over 1 GB**: Mapika/decider-0.8b and 2b, AndeyTait/JevForge-0.8B, IamBusy/OpenJev-0.6B. decider-2b is the best supported of these (20k downloads a month) if a larger chooser is ever needed on Mac. diff --git a/README.md b/README.md index 2aba007..f804dc3 100644 --- a/README.md +++ b/README.md @@ -138,6 +138,17 @@ reason about dropdown options, or write free text. The harness handles observation, typing, selection, and an answer sheet; uploads and essays are left to the person. Submit is never clicked unless enabled. +## Models + +[Documentation/Models.md](Documentation/Models.md) lists the three converted models and +open-weight candidates worth converting next, including +sub-1B-parameter candidates from the Jev Decision Index. +[Documentation/GameDemos.md](Documentation/GameDemos.md) lists game demos, the matched-seed Flappy Bird +comparison, and trial results for Flappy Bird, Connect Four, and Snake (no model +played any of them usefully). Real-time control games are ruled out; the list is +now led by forgiving merge puzzles (drop-merge, Threes) and Codenames, with +Minesweeper kept as a calibration benchmark. + ## License Apache 2.0. CUA-S1-FORMS is MIT, from Cua.