Skip to main content
Speculative routing starts a likely provider request before the selector finishes. On follow-up turns, the host starts the previously persisted compatible route while choosing the route for the next turn. This removes selector latency from the critical path. It requires persistent conversation state and careful ownership of execution, cancellation, and recovery.
Dari’s managed platform implements this by default. The createRouter() handler is selector-first — use the phased API for custom speculation.

​
Why The API Is Phased

prepareRoute() runs deterministic work before any selector call — returning eligible candidates, the recovered previous decision, cost evidence, and the selector request. finalizeRoute() parses the selector’s answer and validates it against the prepared candidates. Keeping these separate lets a host start other effects while the selector runs. The convenience route(input, selector) composes both and is selector-first.

​
Orchestration Sketch

startModel, callSelector, serveOrRecover, and persistNextDecision are host-owned functions:
The serving rule is a host decision. Dari serves the compatible incumbent and stores the selector result for the next turn.

​
When To Speculate

Only when the recovered model and reasoning effort still appear in the prepared candidates — confirming the request’s images, tools, structured output, stream mode, and routing rules still permit it. First turns and stale/incompatible follow-ups remain selector-first.

​
State Between Turns

Router Core does not persist state. The host stores the served model, selector’s next decision, prefix hashes, and provider-cache observations. On the next request, resolve those into previousDecision, chainsByModel, and prefixHits for prepareRoute(). The serving decision and next decision are different values under speculation. Reporting only the speculative incumbent as the selector’s choice would corrupt the route used on the next turn.

​
Failure Semantics

  • Speculative setup fails → wait for selection, execute the selected route
  • Selector fails after incumbent starts → decide whether to continue and what to persist
  • Client disconnects → cancel both, close the stream iterator
  • Don’t emit a terminal event until routing state is durably recorded
  • Attribute usage to the model that served the turn, not the one selected for next
These behaviors are outside createRouter(). It ships operational options — a pluggable lease store, lifecycle hooks for telemetry and billing events, and opt-in cross-model fallback — but speculative orchestration, durable routing state, and billing systems stay host-owned.