Prompt Caching

Cache-aware routing stops a policy from switching away from a warm model for a tiny apparent saving. Give the policy real prices and the conversation’s recent cache state; it removes switches that do not pay for their cold prefix.

Use It In Your Policy

Use createDariRoutingPolicy, not a plain inline RoutingPolicy:

import { createPiRuntime, createRouter } from "@mupt-ai/dari-router";
import { createDariRoutingPolicy } from "@mupt-ai/dari-router/policy-engine";

const pi = await createPiRuntime({ apiKey: process.env.OPENAI_API_KEY! });

const policy = createDariRoutingPolicy({
  runtime: pi,
  selectorModel: "openai/gpt-5.4-mini",
  selectorContextWindowChars: 400_000,

  // Your per-million-token provider prices, including cached-input prices.
  pricing: (model) => myPricing[model] ?? null,

  // Your observed output averages. Router Core does not provide these.
  averageOutputTokensByModel: myAverageOutputTokensByModel,

  // Read the state saved for this conversation after earlier provider calls.
  state: async ({ request }) =>
    request.cacheKey ? cacheState.read(request.cacheKey) : {},
});

const router = createRouter({
  models: [pi.model("openai/gpt-5.4-mini"), pi.model("openai/gpt-5.4")],
  policy,
  executor: pi,
});

cacheState.read() returns the state the policy engine needs:

{
  chainsByModel, // prompt-prefix chain visible to each candidate model
  prefixHits,    // recently observed provider prefixes and the model that served them
  nowMs: Date.now(),
}

After each provider call, save the updated prompt chain, provider-prefix observation, and selected model under the same key. That persistence is application code: Router Framework does not store conversations or cache observations for you.

What You Get

For a conversation with a warm incumbent, the policy engine compares each switch’s cold-prefix cost with its expected savings. It automatically removes a switch unless it saves at least 10% on the fixed-turn cost. So when gpt-5.4-mini looks a little cheaper but switching would reprocess a large warm prefix, it is removed before the selector chooses. A genuinely cheaper switch stays available. This protection works only when all of these are true:

The request has a stable prompt_cache_key.
Your saved state identifies the previous model and a still-warm prefix.
You provide normal and cached-input prices plus observed output-token averages for each model/effort.

Router Core does not ship, infer, or persist output averages. Self-hosted users own those estimates; Dari’s managed platform can maintain its own. If an input is missing or stale, the engine makes no cache-savings claim and does not prune on cache cost.

Send The Same Key Every Turn

Send prompt_cache_key on every request in one conversation:

{
  "model": "my-router",
  "prompt_cache_key": "conversation-42",
  "messages": [{ "role": "user", "content": "Continue." }]
}

Both the OpenAI Chat Completions and Anthropic Messages adapters accept this field. They normalize it to request.cacheKey; createPiRuntime() passes it to Pi as the provider session ID. Custom executors receive request.cacheKey and should map it to the cache or session field supported by their provider. Provider cache hits remain the provider’s decision. Executors return observed usage as cacheReadTokens and cacheWriteTokens; the protocol adapters serialize those values in the caller’s native usage format.

Leases Are A Simpler Fallback

A lease keeps later turns on one model, but it is not a cache-cost calculation or a cache hit. Use leaseTurnsRemaining when you only want short-term stickiness. Use cache-aware routing when you persist prefix observations and want the policy to decide whether a switch is actually worth it. Read Policies for the full policy setup and Decision Pipeline if you need to control persistence and selection phases yourself.

Get Started

Build A Router

Advanced

Prompt Caching

Use It In Your Policy

What You Get

Send The Same Key Every Turn

Leases Are A Simpler Fallback

​Use It In Your Policy

​What You Get

​Send The Same Key Every Turn

​Leases Are A Simpler Fallback

Use It In Your Policy

What You Get

Send The Same Key Every Turn

Leases Are A Simpler Fallback