Skip to content

Replace degenerate agent output with a notice in Slack - #1691

Draft
knechtionscoding wants to merge 1 commit into
kelos-dev:mainfrom
datagravity-ai:fix/slack-degenerate-output-guard
Draft

Replace degenerate agent output with a notice in Slack#1691
knechtionscoding wants to merge 1 commit into
kelos-dev:mainfrom
datagravity-ai:fix/slack-degenerate-output-guard

Conversation

@knechtionscoding

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

When an agent's final message is unusable, Kelos currently pastes it into the
Slack thread verbatim. On 2026-08-21 that put this into a live customer-issue
thread:

`stale=False</li>
</ul>
</section
</section>

FormatSlackTransitionMessage now reads the degenerate marker that
kelos-capture sets in the Task's results and renders a short notice in place
of the response:

⚠️ The model returned an unusable reply. This ran twice and produced
unusable output both times, so there is no answer to show.

The notice is phase-aware — the failed phase says the run was attempted twice
(backoffLimit is 1, so a Failed Task has exhausted both attempts); other
phases get the shorter form without the count claim.

Everything else in the message is left alone: the PR link block still renders
(the agent may well have opened a real PR before its final message degenerated,
and that link is the most useful thing in the reply), the task-name context
block still renders, and the :warning: *Error:* line still renders on the
failed phase.

The substitution happens on the decoded response before both responseToBlocks
and buildFallbackText run, so the fragments stay out of the message Text too
— otherwise they would still ship in notifications and accessibility surfaces
even with the blocks replaced.

The raw response key is left untouched in task.Status.Results. It is the
only evidence available for calibrating the detection thresholds, and nothing
here clears it.

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

Depends on fix/detect-degenerate-claude-output, which introduces the
degenerate result key. Until that lands this branch is inert: with the marker
unset, every existing code path is unchanged. Two of the new tests assert
exactly that (marker absent, and marker present but not "true").

Deliberately narrow:

  • Slack only. TaskReporter (GitHub comments and check runs) is a different
    surface with different readers and is untouched.
  • The progress path is untouched. FormatProgressMessage renders in-flight
    snapshots from pod logs, not the final result, and has no results map.
  • No change to the retry count. backoffLimit stays at 1.
  • The heuristic is not re-derived here. This layer is a marker lookup. The
    reporting package does not have num_turns anyway, and a second copy of the
    predicate would drift from the first.

New tests in internal/reporting/slack_test.go use the real 2026-08-21 garbled
output as the fixture and assert the fragments appear in neither the rendered
blocks nor the fallback text: substitution on succeeded, PR link survival,
notice plus error line together on failed, a degenerate attempt that produced
no response at all, a long garbled response staying a single message (the
multi-message split path is not entered and the block count stays far under
SlackBlockLimit), and the two no-op cases above.

make test and make verify both pass.

Does this PR introduce a user-facing change?

Slack replies for tasks whose agent returned unusable output now show a short notice instead of the garbled text. The pull request link, task name, and error details are still shown.

When capture flags a run's final message as degenerate, Slack now renders
a short notice in place of the unusable fragments rather than pasting them
into the thread. The PR link, the task context block, and the error line on
the failed phase are all preserved, and the raw response is left untouched
in the Task's results.

The substitution happens before both the response blocks and the fallback
text are built, so the fragments stay out of notifications and
accessibility surfaces as well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant