Skip to content

Pull requests: huggingface/lighteval

New pull request New
93 Open 649 Closed
93 Open 649 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

chore: bump doc-builder SHA for main doc build workflow
#1215 opened Apr 16, 2026 by rtrompier Contributor • Draft
1
Add LICA-Bench: graphic design VLM evaluation (39 tasks, 7 domains)
#1212 opened Apr 15, 2026 by purvanshi Loading…
3 of 4 tasks
1
POLLUX LLM-Judge metric
#1210 opened Apr 10, 2026 by ulyanaisaeva Loading…
catch task has no docs instead of throw
#1207 opened Apr 8, 2026 by BuiHoangTu Loading…
add multilingual flag to vllm
#1206 opened Apr 8, 2026 by BuiHoangTu Loading…
fix(vllm): Enhance VLLMModel context size handling for batch inputs
#1205 opened Apr 6, 2026 by paulovsantanas Loading…
1
examples: add RAIL Score responsible AI custom task template
#1203 opened Apr 2, 2026 by SumitVermakgp Loading…
Add --load-tasks-multilingual and fix --custom-tasks for inspect backend
#1199 opened Mar 25, 2026 by dzautner Loading…
4 tasks done
[Bugfix] Check all responses when n>1 instead of only the first one
#1197 opened Mar 23, 2026 by eldarkurtic Contributor Loading…
[Litellm Enhancement] Enable extra sampling args for litellm backend
#1195 opened Mar 20, 2026 by eldarkurtic Contributor Loading…
[Bugfix] presence_penalty is silently dropped from sampling args in litellm backend
#1193 opened Mar 18, 2026 by eldarkurtic Contributor Loading…
[Bugfix] litellm backend should iterate over docs in a split not entire dataset
#1192 opened Mar 18, 2026 by eldarkurtic Contributor Loading…
Remove deprecated prompt_token_ids wrapping in vLLM backend
#1191 opened Mar 18, 2026 by sihyeonn Loading…
Fix litellm connection pool limiting concurrent_requests
#1190 opened Mar 18, 2026 by sihyeonn Loading…
feat(utils): show count of evaluated samples in Markdown summary table
#1188 opened Mar 13, 2026 by anzzyspeaksgit Loading…
1
Fix typos in math_comparison.py and sample_comparison.py
#1186 opened Mar 12, 2026 by joshuaswanson Loading…
squad_v2: include unanswerable questions in evaluation
#1185 opened Mar 9, 2026 by Matteovanypersele Loading…
1
Update vllm version requirement to 0.17.0
#1183 opened Mar 9, 2026 by NathanHB Member Loading…
1
Fail fast on non-retriable LiteLLM status codes
#1182 opened Mar 8, 2026 by yangbaechu Loading…
Redact model config credentials in saved and returned results
#1181 opened Mar 8, 2026 by yangbaechu Loading…
fix(normalizations): guard against index out of range in LogProbToken…
#1180 opened Mar 6, 2026 by inakiLakunza Loading…
Korean completed and Basque fixed
#1179 opened Mar 6, 2026 by inakiLakunza Loading…
[LiteLLM] Add cross-provider reasoning_effort support (first step) + token budget fixes
#1178 opened Mar 5, 2026 by dyurchenko98 Loading…
1
Fix LiteLLM split iteration in greedy_until to avoid duplicate API requests
#1177 opened Mar 5, 2026 by dyurchenko98 Loading…
1
Fix corpus reference orientation for chrF/chrF++/TER metrics
#1176 opened Mar 5, 2026 by dyurchenko98 Loading…
1
1
ProTip! Adding no:label will show everything without a label.