Routing and price options

​

How an endpoint is chosen

• Endpoints that are up come first. Degraded endpoints are only used when no endpoint that’s up can take the request.
• Traffic is shared by capacity. An endpoint with more slots receives a proportionally larger share of requests.
• Similar requests stick together. Requests with the same prompt prefix go to the same endpoint, so the provider’s prompt cache is reused. See Prompt caching.
• Busy endpoints are skipped. If an endpoint is near capacity, the request goes to the next choice instead of queueing.

​

Failover

• A request is tried on up to three endpoints, and no new attempt starts more than 30 seconds after the request arrived.
• Once output has started streaming, the request stays on that endpoint. If the stream breaks after that, it isn’t retried.
• Failover stays within your key’s scope and your price options.
• Pinned requests never fail over.

​

Prompt caching

• For chat completions, the tools, the system and developer messages and the first user message.
• For text completions, the start of the prompt.

{
  "model": "<model-slug>",
  "stream": true,
  "prompt_cache_key": "conversation-8f2c",
  "messages": [{ "role": "user", "content": "Summarize our last exchange." }]
}

​

Price options

{
  "model": "<model-slug>",
  "stream": true,
  "messages": [{ "role": "user", "content": "Hello" }],
  "provider": {
    "max_price": { "prompt": 0.5, "completion": 1.5 },
    "sort": "price"
  }
}

• If no endpoint fits within max_price, the request fails with 400.
• Failover never moves a request above your ceiling.
• You’re always billed at the prices of the endpoint that served the request.

​

Use price options with an OpenAI SDK

stream = client.chat.completions.create(
    model="<model-slug>",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
    extra_body={"provider": {"max_price": {"prompt": 0.5}, "sort": "price"}},
)

const stream = await client.chat.completions.create({
  model: "<model-slug>",
  messages: [{ role: "user", content: "Hello" }],
  stream: true,
  // @ts-expect-error TokenChange extension
  provider: { max_price: { prompt: 0.5 }, sort: "price" },
});