Fix LMEval ONNX evaluator device assignment with recent lm-eval

[justinchuby]

Problem

get_model("ortgenai") / get_model("ort") based evaluation crashes with recent lm-eval versions (reproduced on lm-eval==0.4.12):

AttributeError: property 'device' of 'LMEvalORTGenAIEvaluator' object has no setter

Root cause

lm-eval's LM base class exposes device as a read-only property (backed by self._device). Olive's ONNX evaluators in olive/evaluator/lmeval_ort.py assign self.device = ... in __init__:

• LMEvalORTEvaluator.__init__ — self.device = "cuda" if ep == "CUDAExecutionProvider" else "cpu"
• LMEvalORTGenAIEvaluator.__init__ — self.device = device

With the read-only property inherited from lm-eval, both assignments raise AttributeError, so evaluation never starts.

Fix

Add a device property with a setter to the shared LMEvalOnnxBase (backed by _device), overriding lm-eval's read-only property. The existing self.device = ... assignments keep working across lm-eval versions, and reads of self.device (used for IOBinding placement and .to(self.device)) are unchanged.

Testing

• Added test/evaluator/test_lmeval_ort.py (guarded by pytest.importorskip("lm_eval")) verifying the device property has a setter and round-trips. Both tests pass.
• Verified end-to-end: ran MMLU-Pro evaluation through get_model("ortgenai") on a CUDA INT4 ONNX model (Gemma 4 12B) — evaluation now runs to completion and reports a score (previously crashed at model init).
• ruff check clean on the changed files.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

This PR restores compatibility between Olive’s lm-eval ONNX backends and newer lm-eval releases where LM.device is a read-only property, by overriding device on Olive’s shared ONNX lm-eval base class with a setter-backed property.

Changes:

• Add a device property (getter + setter backed by _device) to LMEvalOnnxBase to avoid AttributeError on self.device = ... assignments.
• Add regression tests to ensure the device property has a setter and that it correctly round-trips stored values.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
olive/evaluator/lmeval_ort.py	Overrides lm-eval’s read-only device property in LMEvalOnnxBase by adding a setter backed by _device.
test/evaluator/test_lmeval_ort.py	Adds regression tests verifying device has a setter and that the property stores/returns values correctly.

[justinchuby]

@copilot fix lint

[Copilot]

@copilot fix lint

Fixed in the latest commit. Added # pylint: disable=wrong-import-position to the import that follows pytest.importorskip("lm_eval") — the guard must precede the import because lmeval_ort.py imports lm_eval at module load time, so the pylint disable is the appropriate fix here. All linters pass clean.