fix(evaluation): do not score LCB samples 0.0 when the judge cannot run

[liayan]

evaluate_livecodebench_worker swallows every exception and returns False, so import/setup failures get scored as wrong answers with subset status "ok". That currently always happens in this env: lcb_runner.prompts imports anthropic (not installed) and datasets>=3 cannot load the script-based code_generation_lite dataset. Observed: 335 samples "graded" 0/335 in 83s; the same outputs score 297/349 through lcb-service. A fake zero on 349/4388 samples is enough to drop a passing DeepSeek-R1 run below the MLPerf accuracy threshold.

• re-raise ImportError/FileNotFoundError/RuntimeError from the worker and the gather loop; the runner already marks the subset failed and sets complete=false
• call load_lcb_benchmark() once before spawning workers so setup errors fail fast with the real traceback

Sample-level grading failures (bad generated code, timeouts) still score as incorrect. This does not fix the local path itself (deps are still missing); it just stops it from reporting a wrong score.

What does this PR do?

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@111cf76). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #432   +/-   ##
=======================================
  Coverage        ?   80.04%           
=======================================
  Files           ?      144           
  Lines           ?    19631           
  Branches        ?        0           
=======================================
  Hits            ?    15713           
  Misses          ?     3918           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arekay-nv]

Thanks for this.
Can you please add some tests? The one thing to look for is whether a single bad example (one that has a bad import) will cause the entire run to fail. I believe that is the reason for swallowing the exception and just counting it as a failure.

[liayan]

Thanks for this. Can you please add some tests? The one thing to look for is whether a single bad example (one that has a bad import) will cause the entire run to fail. I believe that is the reason for swallowing the exception and just counting it as a failure.

Thanks — tests added in 7f391fb.

On the single-bad-example case: generated code never runs in the worker process, so it can't raise ImportError there. evaluate_livecodebench passes it to the LCB grader, which executes it in its own sandboxed subprocess. A bad import in a sample comes back as a failed test result and gets scored 0, same as before. The except Exception: return False in the worker is kept for exactly that case.

The three re-raised types can only come from the judge env itself: ImportError from the lcb_runner imports (lcb_runner.prompts imports anthropic), FileNotFoundError from the missing submodule check, RuntimeError from datasets>=3 refusing to load code_generation_lite. These hit every sample or none — that's how 335/335 got scored 0.0 in 83s. The new pre-flight load_lcb_benchmark() call catches most of this before workers even start.

Tests:

1. worker returns (qid, False) on a generic exception — run continues
2. worker re-raises each of the three infra types
3. one failing sample scores 0, the other samples still grade
4. all workers hitting an infra error raises instead of returning a fake 0/N
5. broken load_lcb_benchmark raises before the pool is created

1 and 3 also pass on main, so the swallow behavior is unchanged. The pool is stubbed with an in-process fake so the tests don't depend on the multiprocessing start method.