Skip to content

Require feedback before disabling Smart Router in Isaac - #1139

Open
JJ27 wants to merge 4 commits into
mainfrom
JJ27/smart-router-off-feedback
Open

JJ27 wants to merge 4 commits into
mainfrom
JJ27/smart-router-off-feedback

Conversation

@JJ27

@JJ27 JJ27 commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

What did you change, and why?

Require a reason before turning Smart Router off in Isaac sessions with ISAAC_SMART_ROUTER_FEEDBACK=1. The reason must contain at least 16 characters after trimming surrounding whitespace. Cancel or Escape keeps routing on. Each new on-to-off transition requires feedback.

The native Claude form validates locally, then disables routing and passes the exact user text to /isaac-feedback --category smart-routing. Opening or cancelling the form makes no model calls. If the command or form is unavailable, routing stays on. Once a valid reason is accepted, a submission failure is reported and leaves routing off.

The CLI validates the same minimum through --smart-router-feedback; the bundled Claude/Codex skill asks for the reason and handles submission. Routing toggles preserve Isaac's rollout flag. Sessions outside the rollout retain their existing behavior.

Requires the shared session flag and argument handling in https://github.com/databricks-eng/universe/pull/2806823.

How do you know it works?

  • 211 focused Python checks passed for session controls, packaged mods, and routing configuration. The required-reason regression fails against the previous implementation.
  • Claude Code 2.1.290 plugin validation and native component checks passed, including 13 TypeScript cases. Coverage includes the 15/16-character boundary, surrounding whitespace, Unicode, exact text, cancellation, repeated toggles, and failures.
  • Ruff check, Ruff format check, and ty passed.
  • A native Claude Code 2.1.290 terminal check passed: Escape left routing on without submission, 15 characters were rejected, 16 were accepted and delivered unchanged, and re-enabling then disabling required a fresh reason. It verified the real CLI session controls and exited normally. The receiver was local; this does not establish a live evidence upload or macOS coverage.

What it looks like

smart-router-feedback-pr1139-color.mp4

Comment thread skills/smart-router/SKILL.md Outdated
skill to further work only if the session was launched with `ENABLE_SMART_ROUTER_ORCHESTRATOR=1`.
Do not change the orchestration feature flag. Return the command's result.
If the result includes the optional `/isaac-feedback` invitation, show it at most
once in this session. Do not wait for feedback or submit it automatically.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lilly-luo currently set to 'at most once per session', should we do this every time smart router is manually toggled off?

@JJ27
JJ27 marked this pull request as ready for review October 15, 2026 22:27
Copilot AI balanced review requested due to automatic review settings October 15, 2026 22:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JJ27
JJ27 requested review from lilly-luo and rohita5l October 15, 2026 22:59
@JJ27
JJ27 force-pushed the JJ27/smart-router-off-feedback branch from 9f83e04 to 6151370 Compare October 16, 2026 03:10
Copilot AI balanced review requested due to automatic review settings October 16, 2026 03:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JJ27
JJ27 force-pushed the JJ27/smart-router-off-feedback branch from 6151370 to e775b9c Compare October 16, 2026 17:07
Copilot AI balanced review requested due to automatic review settings October 16, 2026 17:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JJ27
JJ27 force-pushed the JJ27/smart-router-off-feedback branch from e775b9c to 893c521 Compare October 16, 2026 19:17
Copilot AI balanced review requested due to automatic review settings October 16, 2026 19:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JJ27 JJ27 changed the title Offer optional feedback after disabling Smart Router Require feedback before disabling Smart Router in Isaac Oct 16, 2026
Copilot AI balanced review requested due to automatic review settings October 16, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants