Skip to content

Show when unit test output is cut, and keep its end - #331

Open
gorandodig wants to merge 1 commit into
mainfrom
feat/eng-324-visible-test-output-budget
Open

gorandodig wants to merge 1 commit into
mainfrom
feat/eng-324-visible-test-output-budget

Conversation

@gorandodig

@gorandodig gorandodig commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Rewritten files come back without a final newline, so linter tests in unit-test scripts never converge.

When a unit test script fails, its output goes to the fixer, which works with at most 10,000 characters. Output over that is now cut on the client: the fixer gets the first and the last ~4,980 characters, with a marker in between that says how much was left out ([... 952 characters omitted ...]). Each time, a warning shows in the TUI, on headless stdout and in codeplain.log, without --verbose:

WARNING codeplain: Unit test output is 10,912 characters, over the 10,000-character budget. The fixer gets the start and the end; 952 characters in the middle are left out.

The budget, and how to stay within it, is documented for test script authors in docs/starting_a_plain_project_from_scratch.md and in the --unittests-script help (also in docs/plain2code_cli.md).

Please review this behavior change: because the client now keeps the output within the server's budget, the server no longer summarizes long unit test output with an LLM. The fixer sees the script's own text, start and end. One difference: the server collapses repeated lines before it measures, while the client measures the raw output, so output that only fit after collapsing is now cut. Conformance test output is unchanged: the client does not cut it, and the server still summarizes it.

Verification

  • Ran FixUnitTests on 10,912 characters of output through the real setup_logging (headless, logging to file, no --verbose): the warning above appeared on stdout and in the log file.

Summary by CodeRabbit

  • Bug Fixes
    • Long unit-test failure output sent for fixing is now limited to 10,000 characters while preserving the beginning and end. A warning reports how much was omitted.
  • Documentation
    • Updated the CLI help and project setup guide with details about output limits and recommendations for keeping test and lint output concise.

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3cfa7df6-d9ac-4704-88d4-6736f6285e74
📥 Commits

Reviewing files that changed from the base of the PR and between 2f721a4 and 4962e05.

📒 Files selected for processing: 7
  • docs/plain2code_cli.md
  • docs/starting_a_plain_project_from_scratch.md
  • plain2code_arguments.py
  • plain2code_utils.py
  • render_machine/actions/fix_unit_tests.py
  • tests/test_fix_unit_tests_output_budget.py
  • tests/test_plain2code_utils.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Unit-test output sent to the fixer is now capped at 10,000 characters by omitting the middle and retaining the beginning and end. The change adds truncation tests and updates CLI and project setup guidance.

Changes: Unit-test output budget

Layer / File(s) Summary
Middle truncation utility and tests
plain2code_utils.py, tests/test_plain2code_utils.py
Adds truncate_middle, which returns text within the budget unchanged or replaces the middle with an omission marker. Tests cover retained text, omitted counts, and budgets that cannot fit the marker.
Fixer integration and output guidance
render_machine/actions/fix_unit_tests.py, tests/test_fix_unit_tests_output_budget.py, plain2code_arguments.py, docs/plain2code_cli.md, docs/starting_a_plain_project_from_scratch.md
Applies the 10,000-character limit before sending unit-test output to the fixer and logs truncation details. Tests verify the sent output and warning. The CLI help and project guide describe the limit and output recommendations.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: nejcs, pedjaradenkovic

Merge Risk: ⚪ Minimal · up to 4962e

The unit-test output limit appears ready to merge after normal checks.

Security architecture risk: 🔵 Low · up to 4962e

The change limits submitted test output without adding access or permissions. No introduced security issue was established, but service-side handling of the shortened output remains unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated exposure remains the current rendering context's diagnostic submission and existing response-file application path. The change adds no recipient, credential, or client-side file authority; broader service or tenant isolation cannot be assessed from the available consumer evidence.

Trust Boundaries and Controls

  • observed — Truncation is a length control, not sanitization: retained text is copied directly from script output. It does not neutralize instructions or assign greater authority to that text. Whether server summarization had any security-control role remains unverified.

Resilience and Maintainability Implications

  • observed — The transformation leaves the original action payload intact and produces the same result for repeated input. API failure still precedes local file application. Response application and subsequent tracking retain their existing ordering; sequential-write interruption and recovery risks predate this change.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 20.83% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 24 functions across 5 files. (2 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: truncating oversized unit-test output while preserving its end.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 20.83% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 24 functions across 5 files. (2 skipped: 2 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Unit test output over the fixer's 10,000-character budget was handed to
the server whole, which summarized it with an LLM (or kept the first 85%)
and left only a DEBUG line behind. Lint findings and summary lines, which
scripts print last, were the part most often lost, and script authors
could not see that it happened.

The client now cuts output over the budget from the middle, keeping equal
shares of the start and the end with a marker saying how much was left
out, and logs a WARNING with the original size and the budget. The budget
and how to stay within it are documented for test script authors.
@gorandodig gorandodig closed this Oct 14, 2026
@gorandodig
gorandodig deleted the feat/eng-324-visible-test-output-budget branch October 14, 2026 10:27
@gorandodig
gorandodig restored the feat/eng-324-visible-test-output-budget branch October 14, 2026 10:28
@gorandodig gorandodig reopened this Oct 14, 2026
@gorandodig
gorandodig force-pushed the feat/eng-324-visible-test-output-budget branch from 73a4bea to 4962e05 Compare October 14, 2026 10:29
@gorandodig
gorandodig marked this pull request as ready for review October 14, 2026 14:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant