Inference errors - TokenChange

Errors use the OpenAI error format. Match on the HTTP status and code; messages can change.

{
  "error": {
    "message": "Invalid API key",
    "type": "invalid_request_error",
    "code": "invalid_api_key"
  }
}

Error reference

Status	`code`	Cause	Retry
`400`		The request is invalid: missing `model`, `stream` not `true`, `max_tokens` below 1, `temperature` outside 0 to 2, invalid `provider` options, or invalid JSON	No. Fix the request.
`400`		No endpoint fits your `provider.max_price`	No. Raise the ceiling or remove it.
`400`		The endpoint rejected the request, for example because it’s too long for the context window or uses a field the endpoint doesn’t support	No. Change the request or the model.
`401`	`invalid_api_key`	The key is missing, invalid, revoked or expired, or you sent a REST API token	No. Use a valid inference key.
`402`	`insufficient_credits`	Your balance is zero	After adding credits.
`403`	`account_suspended`	Your account is suspended	No. Contact support.
`403`	`insufficient_permissions`	Your key is limited to endpoints that don’t serve this model	No. Use a key with the right scope.
`404`	`not_found`	The model doesn’t exist, or none of its endpoints is serving	Later, or use another model.
`429`	`rate_limit_exceeded`	Every eligible endpoint is at capacity, or the endpoint rate-limited the request	Yes, after `Retry-After`.
`502`	`upstream_error`	Every attempt to reach an endpoint failed	Yes, with backoff.
`502`		The endpoint ended the stream before sending anything	Yes, with backoff.
`503`	`service_unavailable`	Every eligible endpoint is recovering from recent failures	Yes, after `Retry-After`.
`500`		Internal error	Yes, with backoff.

Error messages from endpoints are never passed through. When an endpoint fails, you get a generic message such as upstream request failed.

Honor the Retry-After header on 429 and 503. It’s in seconds.
For 502 and 500, retry with exponential backoff.
TokenChange already retries on other endpoints before returning an error, so retry fewer times than you would against a single server.
Pinned requests aren’t retried on other endpoints. Remove the pin to let TokenChange fail over.

Failed requests that delivered no output aren’t billed.

Once the stream has started, the status is already 200. A failure then arrives as an error event followed by data: [DONE]:

data: {"error":{"message":"upstream stream interrupted","type":"server_error"}}

Check each event for an error field. Output delivered before the error is billed.