Repository navigation
Size the judge's response budget for reasoning models and keep the CLI prompt off argv - #106
Merged
Merged
Conversation
…I prompt off argv The response budget was fixed at 500 tokens with no way to change it. That fits the judge's JSON answer, but reasoning models (OpenAI o-series and gpt-5, GLM, Kimi, DeepSeek) think inside the same budget first, so they returned empty content and the user saw "no valid JSON object found in response:" with nothing after the colon. Fixes #103. Default the budget to 8192 tokens, expose it as --max-response-tokens and judge.DefaultMaxResponseTokens, and raise the HTTP timeout from 30 seconds to 3 minutes to match. When a response carries no answer text, say why: both clients now return ErrEmptyResponse, and when the finish reason or the presence of reasoning_content, reasoning, or thinking blocks shows the budget was used up thinking, the message names the budget and the flag. The Anthropic client picks the first text block rather than the first block, so a gateway with extended thinking on no longer returns the thinking block's empty text. The claude-cli provider passed the system prompt and the content as arguments. The judge prompt alone is about 24 KB, so with the content the command line exceeded the Windows limit of 32 KB, or 8 KB through a .cmd shim. The content now goes to the CLI on stdin and the system prompt through a temporary --system-prompt-file, available since Claude Code 1.0.55; the file is removed after the call. Verified against the installed CLI, which returned the full six-dimension score.
The paragraph was inserted at the top of the README instead of after the --max-tokens-style table it refers to.
There was a problem hiding this comment.
🟡 Changes recommended
Empty Anthropic text blocks still bypass the new empty-response and budget diagnostics.
1 open issue
What changed in this PR
Addresses #103 by increasing the judge’s reasoning budget and keeping large Claude CLI prompts off command-line arguments.
Changes:
- Adds an 8,192-token default, configurable response budget, and longer HTTP timeout.
- Improves empty-response diagnostics and Anthropic text-block selection.
- Moves CLI prompts to stdin and temporary files, with tests and documentation.
| File | Description |
|---|---|
| README.md | Documents budgets, diagnostics, and CLI prompt transport. |
| judge/client.go | Updates budgets, response handling, and CLI invocation. |
| judge/client_test.go | Tests response handling, budgets, and prompt transport. |
| cmd/score_evaluate.go | Adds and validates the response-budget flag. |
| cmd/score_evaluate_integration_test.go | Checks flag visibility and zero-value rejection. |
Review effort: Balanced
The Anthropic client returned the first text block even when it was empty, so a thinking block followed by an empty text block at stop_reason max_tokens came back as "" with no error, skipping the budget diagnostic, and an empty text block hid any later text block. An empty content array also exited before the diagnostics. Flagged in review. Return the first text block that has text; otherwise route every no-answer shape through the same check, naming the budget when the stop reason is max_tokens and ErrEmptyResponse otherwise.
dacharyc
force-pushed
the
fix/issue-103-response-budget
branch
from
October 18, 2026 16:04
b4436b4 to
195e179
Compare
OpenAI's own reasoning models do not return their thinking in the response, only its size in usage.completion_tokens_details. Read it, so an empty answer from them names the budget and says how many tokens went to reasoning, rather than relying on the finish reason alone. The claude-cli provider returned empty output as a successful empty string, which failed later in the JSON parser with nothing to show. Return ErrEmptyResponse instead; the CLI manages its own token budget, so no budget hint applies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #103.
What was wrong
judge.NewClientfixed the response budget at 500 tokens with no flag to change it. The judge's JSON answer fits in that, but reasoning models (OpenAI o-series and gpt-5, GLM, Kimi, DeepSeek) think inside the same budget before answering, so they returned empty content and the user sawno valid JSON object found in response:with nothing after the colon.claude-cliprovider passed the system prompt and the content as command-line arguments. The judge prompt alone is about 24 KB, so with the default 8 KB of content the command line sat at the Windows limit of 32 KB, and far over the 8 KB limit that applies whenclaudeis launched through a.cmdshim. Not only--full-contentfailed; the default did.What changed
judge.DefaultMaxResponseTokensis 8192, used whenClientOptions.MaxResponseTokensis zero.score evaluateexposes it as--max-response-tokens, validated before an API key is looked up. The HTTP timeout goes from 30 seconds to 3 minutes, since a reasoning model can take that long on a large budget.judge.ErrEmptyResponsewhen a successful call carries no answer text. When the finish reason, the presence ofreasoning_content,reasoning, or Anthropic thinking blocks, or a nonzerousage.completion_tokens_details.reasoning_tokens(OpenAI's own reasoning models keep the text private) shows the budget was used up thinking, the message names the budget and the flag, with the reasoning token count when it is reported. Empty output from theclaudeCLI is also reported as an empty response rather than failing later in the JSON parser. The Anthropic client now takes the firsttextblock rather than the first block, so a gateway with extended thinking on no longer hands back a thinking block's empty text.--system-prompt-file(Claude Code 1.0.55 and later), removed after the call.buildArgsno longer carries prompt text at all. The CLI is invoked through a package variable so tests can capture the invocation.claude-cli.Verification
score evaluate --helplists the flag;--max-response-tokens 0is rejected with a clear message before any key is needed.claudeCLI (2.1.292) on a scratch copy of the valid-skill fixture: a full six-dimension score came back in about five seconds and no prompt file was left in the temp directory.go test -race ./...,go vetfor darwin and windows, andgolangci-lint runare clean.No CHANGELOG entry, following the release-commit pattern.