chore: reverts judge message stripping#205
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2e40f00. Configure here.
| (model, provider, messages, instructions, | ||
| tracker_factory, enabled, judge_configuration, variation) = self.__evaluate( | ||
| key, context, default.to_dict(), extended_variables | ||
| key, context, default.to_dict(), variables |
There was a problem hiding this comment.
Judge placeholders cleared at config
High Severity
_judge_config passes user variables into __evaluate, which Mustache-interpolates every message. Legacy judge templates like {{message_history}} and {{response_to_evaluate}} are not in that map, so chevron blanks them before Judge._construct_evaluation_messages runs, breaking legacy judge evaluations loaded from flags.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2e40f00. Configure here.
|
|
||
| evaluations_task = await self._track_judge_results(tracker, input_text, result.content) | ||
|
|
||
| self._messages.append(assistant_message) |
There was a problem hiding this comment.
Failed run appends assistant history
Medium Severity
ManagedModel.run always appends an assistant LDMessage after the runner returns, even when result.metrics.success is false. A failed call still leaves a user turn plus an empty or error assistant turn in _messages, which pollutes later all_messages sent to the model.
Reviewed by Cursor Bugbot for commit 2e40f00. Configure here.
1 participant
Requirements
Related issues
https://launchdarkly.atlassian.net/jira/software/c/projects/AIC/boards/2045?selectedIssue=AIC-2848
Describe the solution you've provided
This is a reversion of functionality that was added.
Describe alternatives you've considered
Reverts behavior from #165 that stripped messages off of the judges and then re-created them.
Additional context
This was discussed and there are various concerns about this behavior. Since this is a semi-permanent solution once changes are applied on the server side we don't want to take this step yet.
Note
Medium Risk
Judge and managed-completion paths change how prompts are built and sent to providers, which can alter evaluation behavior for legacy judge configs; runner input typing is broader but list-input semantics are a contract change for direct runner callers.
Overview
Reverts the prior judge flow that stripped legacy template messages in
LDAIClientand sent a single formatted string to the model.judge_config()again returns the full message list (including{{message_history}}/{{response_to_evaluate}}placeholders), andJudge.evaluate()builds the prompt by Mustache-interpolating those templates into aList[LDMessage]passed toRunner.run(). Evaluations now fail early when judge config has no messages.The
Runnercontract is extended sorun()accepts either a string (still appended to multi-turn history) or a message list used as the complete payload without prepending runner history or config messages. OpenAI and LangChain factories no longer seed runners withconfig.messages.ManagedModeltakes over completion conversation management: it keeps turn history, prepends config system messages, invokes the runner with the full list, and feeds judges history as CRLF-joined content (notrole:lines).Reviewed by Cursor Bugbot for commit 2e40f00. Bugbot is set up for automated code reviews on this repo. Configure here.