Repository navigation
Run The Sunken Crypt on three decision models, and say what the laten… - #5
Merged
Merged
Conversation
…cies measure godot-jev: compare_backends.gd takes extra Open Jev servers with `--openjev name=url`, records how many options each question offered and the thresholds used, and counts agreement for every pair of backends. The 16 session lines were run for 3 rounds on TypeSafe Jev (jev-1.13.0), Open Jev 27B and Open Jev Flash 9B: all three landed 48 of 48 with no wrong move, and Flash 9B, being less sure, made the game ask on two lines. The run is written up in eval/results-2026-10-08.md and recorded under check 2 in the README. Both result files it rests on are committed, including the 2026-09-30 run it is compared with. docs: a godot-jev results page in the style of the model-router one. Its data comes from docs/tools/build-godot-jev-data.gd, which replays every recorded answer through the game's own parser, at the run's thresholds and at each setting the page's sliders reach, and refuses to write if an outcome or a summary differs from the run. The styles, tooltips, scoreboard and latency chart the two results pages share move to docs/assets/results.css and results.js. Latency: the local models' timings are from 4-bit MLX builds on one laptop running macOS with other applications open, and the hosted ones include the network. The landing page, both results pages and the READMEs now say that these are readings of a setup, not of how fast the models are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
LGTM |
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
⛔ Files ignored due to path filters (1)
📒 Files selected for processing (19)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…cies measure
godot-jev: compare_backends.gd takes extra Open Jev servers with
--openjev name=url, records how many options each question offered and the thresholds used, and counts agreement for every pair of backends. The 16 session lines were run for 3 rounds on TypeSafe Jev (jev-1.13.0), Open Jev 27B and Open Jev Flash 9B: all three landed 48 of 48 with no wrong move, and Flash 9B, being less sure, made the game ask on two lines. The run is written up in eval/results-2026-10-08.md and recorded under check 2 in the README. Both result files it rests on are committed, including the 2026-09-30 run it is compared with.docs: a godot-jev results page in the style of the model-router one. Its data comes from docs/tools/build-godot-jev-data.gd, which replays every recorded answer through the game's own parser, at the run's thresholds and at each setting the page's sliders reach, and refuses to write if an outcome or a summary differs from the run. The styles, tooltips, scoreboard and latency chart the two results pages share move to docs/assets/results.css and results.js.
Latency: the local models' timings are from 4-bit MLX builds on one laptop running macOS with other applications open, and the hosted ones include the network. The landing page, both results pages and the READMEs now say that these are readings of a setup, not of how fast the models are.