MultiTurnRoute Arena

A router picks which LLM answers each turn of a session. The Arena scores a router on recorded and live sessions for answer quality and cost.

  1. Why two tracks
  2. How the tracks work
  3. Leaderboard
  4. Data and rules

Why the Arena Has Two Tracks

Comparing routers is easy on single questions and hard on multi-turn sessions.

The problemEach reply changes what the user asks next.

A single question can be scored ahead of time. A session branches, so it cannot.

The two tracksStatic ranks routers on one fixed set of models. Dynamic tests your own models and sessions.

Static: keep the recorded session

The history holds the session’s recorded replies. Routers are scored turn by turn, offline.

Dynamic: run the session live

The selected model’s reply shapes the next user query. A judge rates each answer in its dialogue context.

How the Tracks Work

Press a step to play it.

Static track

Same 9 models, same recorded sessions

You
Arena

1Collect dataArena

14,966 training turns, every answer graded.

2Train a routerYou

Learn from the training turns, with any method.

3Submit codeYou

# my_router.py
@register_router
class MyRouter(Router):
  def decide(self, turn, state):
    return "gemma-31b"
$ multiturnserve arena submit my_router.py

Packs your router file for the organizers’ replay.

4Replay hidden turnsArena

3,671 hidden test turns.

5Rank routersArena

Ranked by area under the deferral curve (AUDC).

Dynamic track

Your models, sessions run live

You
Arena

1Provide tasksArena

Tasks are your own file in chat.jsonl format; each goal is a conversation’s first message. Test conversations are not public yet.

2Run sessions liveYou

You run the router, the models and the simulated user.

3Judge answersYou

The judge rates each answer in context. For each cost weight, the Arena averages quality and normalized cost over rated turns and sums the bill over all turns.

4Submit logsYou

Pack the run log with arena submit --log.

5CompareArena

Use the same tasks, model pool, and user simulator. Results are self-reported.

Static Track Leaderboard

Official ranks use area under the deferral curve (AUDC) across 3,671 held-out turns and nine models. CoR ranks first with an AUDC of 0.7389.

View all six AUDC scores
  1. Chain-of-Route — 0.7389
  2. RAG history — 0.6998
  3. Single-turn — 0.6963
  4. Bao et al. (simplified) — 0.6909
  5. Full history — 0.6905
  6. kNN — 0.6858

Compare at a chosen cost weight

Move the slider to rank each router at its best point for that weight.

quality onlycost first
#RouterReadsQuality0–1Cost vs gemma-31bScorequality − w × cost

One run (seed 0). *Bao et al. is our simplified reimplementation. The dynamic track has no results yet. The score weight w is not the router’s cost weight λ: λ sets each pick and traces the curve, and w picks a point on that curve afterwards.

Data and Rules

What stays the same for every submission.

Data3,996 conversations (18,637 turns) from WildChat and LMSYS-Chat-1M, answered by 9 models (granite-3b to mistral-medium-128b). Training turns come with every answer’s grade; test-turn grades stay hidden. The data are not public yet.
JudgeAn LLM grades each answer on a 1–9 score, mapped to 0–1: NVIDIA Nemotron 3.5 Super, same prompt in both tracks.
Simulated userIn the dynamic track, an LLM plays the person chatting: DeepSeek-V4.1-Flash-1017, same set-up for everyone.
CostStatic replay tracks reusable prefixes along each router’s chosen path and uses a fixed price table. In live runs, the router estimates prefixes before selection and the bill uses the selected backend’s reported token usage.