Chain-of-Route

Chain-of-Route (CoR) is our routing algorithm in MultiTurnRoute. It predicts response quality from dialogue history and estimates cost from each candidate’s reusable prefix before selecting one model.

HistoryThe Dialogue State Propagator (DSP) combines recent turns with a recurrent summary. A learned gate controls how much history to use for the current query.
Session state
Routerpredicts each model’s quality
KV cacheThe Serving State Tracker (SST) prices the estimated reusable prefix at the cached-input rate. In the WildChat eviction experiment, SST saw each candidate’s surviving prefix before model selection.
KV cache per model
Routerestimates each model’s cost

pick = argmax over models of (1 − λ) × quality − λ × cost / (cost of the most expensive model)

Up to 20.3%lower normalized cost to match the strongest model’s quality, relative to Bao et al. on the paper’s five-model benchmark. CoR matched or improved peak quality (Table 1).
84%of gemma-31b’s cost, at higher quality (0.726 vs 0.706): CoR on the MultiTurnRoute Arena static track, at score weight w = 0.12 (score = quality − w × cost, cost as a share of gemma-31b’s). w is not the router’s λ: λ sets each pick, and w picks a point on the curve that sweeping λ traces. At gemma-31b’s quality, CoR costs 74%. See the leaderboard