A router picks which LLM answers each turn of a session. The Arena scores a router on recorded and live sessions for answer quality and cost.
Comparing routers is easy on single questions and hard on multi-turn sessions.
A single question can be scored ahead of time. A session branches, so it cannot.
The history holds the session’s recorded replies. Routers are scored turn by turn, offline.
The selected model’s reply shapes the next user query. A judge rates each answer in its dialogue context.
Press a step to play it.
Same 9 models, same recorded sessions
14,966 training turns, every answer graded.
Learn from the training turns, with any method.
# my_router.py @register_router class MyRouter(Router): def decide(self, turn, state): return "gemma-31b" $ multiturnserve arena submit my_router.py
Packs your router file for the organizers’ replay.
3,671 hidden test turns.
Ranked by area under the deferral curve (AUDC).
Your models, sessions run live
Tasks are your own file in chat.jsonl format; each goal is a conversation’s first message. Test conversations are not public yet.
You run the router, the models and the simulated user.
The judge rates each answer in context. For each cost weight, the Arena averages quality and normalized cost over rated turns and sums the bill over all turns.
Pack the run log with arena submit --log.
Use the same tasks, model pool, and user simulator. Results are self-reported.
Official ranks use area under the deferral curve (AUDC) across 3,671 held-out turns and nine models. CoR ranks first with an AUDC of 0.7389.
Move the slider to rank each router at its best point for that weight.
| # | Router | Reads | Quality0–1 | Cost vs gemma-31b | Scorequality − w × cost |
|---|
One run (seed 0). *Bao et al. is our simplified reimplementation. The dynamic track has no results yet. The score weight w is not the router’s cost weight λ: λ sets each pick and traces the curve, and w picks a point on that curve afterwards.
What stays the same for every submission.