MultiTurnRoute: A Platform for Building and Evaluating Multi-Turn LLM Routers

MultiTurnRoute gives built-in and custom routers a shared interface for choosing an LLM at each turn. Our router, Chain-of-Route, predicts quality from dialogue history and estimates cost from reusable prefixes. MultiTurnRoute Arena evaluates routers on recorded and interactive conversations.

MultiTurnRoute platform overview. At each turn a registered router selects a model using the query and session state. Chain-of-Route, five baselines, and custom routers share an interface. The Arena evaluates them in static replay and interactive sessions.
Figure 1. The session runtime, router implementations, and evaluation Arena. View full size (PDF)
git clone https://github.com/loopleap-lab/MultiTurnRoutecd MultiTurnRoute && pip install -e ".[serve]"

Why Routing Needs Session State

In a session, the right model for a turn depends on the turns before it.

MeaningA short follow-up can depend on a much earlier reply. The dialogue history helps the router interpret the current query.
“Shorter, please” looks easy alone; after “Summarize this 12-page paper” it is a hard task.
58.7%WildChat
54.2%LMSYS-Chat-1M

of held-out turns had a quality score at least 3 points lower on the 1–10 scale when prior turns were removed (paper, Section 4).

CostA model can reuse a prompt prefix from an earlier turn if it still holds that prefix in its KV cache. Reuse can lower the input cost of choosing that model again.
gemma-31b, price per 1M tokens
Input$0.20
Cached input$0.04
Switching to a model that has not read the session pays for the whole history again.
~95%In the five-model benchmark (Section 4), a cost estimate that ignores the cache is off by about 95% (mean absolute percentage error) at turn 6.

The MultiTurnRoute runtime

Your app sends each turn and a router picks the model. MultiTurnRoute keeps the state of each session.

Your app
Chat assistantsSupport botsResearch assistants
Routers
Use dialogue historyChain-of-Route (ours)Full historyRAG historyBao et al. (simplified)
Use the current querySingle-turnkNN
+ your router

All plug in the same way and can be submitted to MultiTurnRoute Arena.

MultiTurnRoute
New sessionone state per session
Each turnask the router, send the turn to its pick
After the replyrecord usage and update session state
Historywhat each session said
Reusable prefixestimated per model from its last prompt and output token counts
Where the models run
vLLM on your GPUsOpenRouterprovider APIs+ your backend

Step Through a Session

Click Next turn to send the next message of a simulated session, and see which model the router picks, what it costs, and why.

A research assistant, routed by Chain-of-Route

Get Started

Install, start the gateway with the example config, and send a session’s turns through it. No GPU, data or API keys needed.

Show the router file