[iziz]

Problem

run_test_std in llm_evaluation/livecodebench_util.py treats a completion as a script (it prepends __name__ = "__main__" before exec) only when the completion contains the exact string __name__ == "__main__". A program that uses the equally valid single-quoted form

if __name__ == '__main__':
    main()

is exec'd with __name__ bound to builtins, so its entry point never runs, it prints nothing, and every stdin test fails, even when the solution is correct.

Reproduction

from livecodebench_util import run_test_std

body = "def main():\n    print(input()[::-1])\n\n"
print(run_test_std(body + "if __name__ == '__main__':\n    main()\n", "abc\n", "cba"))
# before: (False, '')   after: (True, 'cba')

(run_test_std replaces sys.stdout; restore it before printing the result.) With the current code, the single-quoted guard and the reversed form '__main__' == __name__ fail; the double-quoted guard and code without a guard pass.

Impact

LiveCodeBench rows in router_inference/predictions/ (main, 04c99fd) that were scored 0 and whose last code block uses a single-quoted guard:

prediction file	rows
PSAI-Auto-Router	30
nadir-caliper	25
auto_router	22
sqwish-fvcta8	20
Paix2-router	20
divyam-router	14
BARouter	10
llm-router	10
agentforge-router, hybrid-router	3 each
sqwish-router	2
cross-router, r2-router, lynkr	1 each
KT-ModelRouter, orcarouter-adaptive, azure-model-router, AsiaInfo-Router	0

Open submissions: #229 (BizRouter) has 74 such rows and #228 (dalpha-llm-router) has 7.

To check that these are real solutions and not just a formatting difference, we re-scored the 74 rows of #229 inside the evaluation image with no network: the current code scores all 74 as 0 (reproduced), and with this patch 60 of them pass the tests.

Disclosure: we submitted #229, which is the submission most affected by this bug, because its main model habitually writes the single-quoted guard. The change applies the same rule to every submission; whether and how to re-evaluate existing entries is up to the maintainers.

Fix

Match the guard with either quote style and either operand order (a regex instead of a fixed substring). Completions without a guard, and completions with the double-quoted guard, are executed exactly as before. A guard that can never be true (for example if __name__ == '__other__':) still does not run.

Checked with the pinned pre-commit hooks (ruff v0.11.7, ruff-format, codespell v2.4.1).

[yl231]

Thanks, @iziz, and thanks for disclosing the link to #229. Merged. We checked it with our own test programs: single-quoted guards now run, and code without a guard, double-quoted guards, a guard that can never be true, and wrong answers behave as before. This matches upstream LiveCodeBench, which normalises the guard with ast.unparse. The next full re-grade applies it to every router.