Skip to content

Model router#5324

Open
shatfield4 wants to merge 32 commits intomasterfrom
5313-feat-model-router
Open

Model router#5324
shatfield4 wants to merge 32 commits intomasterfrom
5313-feat-model-router

Conversation

@shatfield4
Copy link
Copy Markdown
Collaborator

@shatfield4 shatfield4 commented Apr 2, 2026

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #5313
Translations PR: #5390

Description

  • Add model router as a new provider type that routes messages to different LLM providers/models based on configurable rules
  • Support two rule types: calculated (deterministic conditions) and LLM-classified (semantic matching via fallback model)
  • Route based on prompt content, conversation token count, message count, time of day, or image attachments
  • CSV support for keyword matching ("python,javascript,coding" matches any)
  • Configurable cooldown to cache routing decisions and avoid reevaluation on every message (decided to keep this so that users can dial it in to work for their specific workflows)
  • Full settings UI for creating/editing routers and rules with drag-and-drop priority ordering
  • Inline provider credential setup when configuring routing targets
  • Works in both regular chat and agent mode, including per-turn re-routing in agent mode
  • Ephemeral routing notification shown in chat during streaming with animated border, disappears on refresh
  • Router selectable at system level or per-workspace via the model picker

Visuals (if applicable)

Model routing animation

model.routing.animation.mov

Chat UI with model router switching to different models

Screenshot 2026-04-08 at 4 06 39 PM

Agent mode chat UI with router switching to different models

Screenshot 2026-04-08 at 4 11 37 PM

Model router settings page

Screenshot 2026-04-08 at 4 07 00 PM

Editing router

Screenshot 2026-04-08 at 4 07 28 PM

Router rules configuration

Screenshot 2026-04-08 at 4 07 17 PM

Additional Information

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

@shatfield4 shatfield4 self-assigned this Apr 2, 2026
@shatfield4 shatfield4 linked an issue Apr 2, 2026 that may be closed by this pull request
@shatfield4 shatfield4 marked this pull request as ready for review April 9, 2026 00:20
@shatfield4 shatfield4 removed their assignment Apr 9, 2026
Copy link
Copy Markdown
Contributor

@angelplusultra angelplusultra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, everything seems to work smoothly.

I dont see this env var updated in the .env.example. It should be grouped

# LLM_PROVIDER="anythingllm-router"
# MODEL_ROUTER_ID=1

Comment thread frontend/src/pages/GeneralSettings/LLMPreference/index.jsx
Comment thread frontend/src/pages/GeneralSettings/LLMPreference/index.jsx
Comment thread server/endpoints/modelRouter.js
Comment thread server/prisma/schema.prisma
Comment thread server/models/modelRouter.js Outdated
Comment thread server/utils/chats/stream.js
Comment thread server/utils/AiProviders/modelRouter/index.js
Comment thread server/utils/router/llmClassifier.js Outdated
Comment thread server/utils/router/llmClassifier.js Outdated
Copy link
Copy Markdown
Contributor

@angelplusultra angelplusultra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

NIT: Not super crucial because I know this is a pre-design implementation but the row on a rule from "{POSITION_NUMBER} {RULE_NAME} {RULE_TYPE}" looks unaligned to me and the rule type is black against dark mode, making it hard to read. I know it's pre-design but just wanted to point it out.

Image

Comment thread server/models/modelRouter.js
Comment thread server/models/modelRouter.js
… support (#5478)

* Implement draft designs

* add matches comparator for prompt content

* implement multi conditional to a calculated rule

* fix between comparator

* restyle model router modals to match light mode designs

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
@shatfield4
Copy link
Copy Markdown
Collaborator Author

shatfield4 commented Apr 22, 2026

  • Convert semantic router custom json tool to an aibitat tool

@shatfield4
Copy link
Copy Markdown
Collaborator Author

shatfield4 commented Apr 22, 2026

@angelplusultra just a couple new changes here:

  • New webm animation and cleanup dead code for old animation
  • Now uses aibitat instead of the custom json tool to classify prompts
  • bug fix with the new rule modal not resetting each time we make a new rule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT]: Model router

2 participants