XCoffee
Back to blog

How to Tell If Your LLM API Relay's Failover Is Real

Nov 11, 2026XCoffee Team

How to check a relay with a controlled failure, response data and a request trace—and why status codes, model names and latency alone cannot prove failover.

How to Tell If Your LLM API Relay's Failover Is Real

Almost every relay and gateway advertises the same sentence: automatic failover on rate limits and outages. That sentence alone does not explain which errors trigger a retry, whether another upstream is tried, or what happens after output starts.

This post gives you a first inspection using a key and a terminal against an OpenAI-compatible endpoint. A short black-box test can reveal symptoms, but proving an upstream switch usually needs a controlled failure and a request trace. You may need the operator’s help. Use a test account, respect rate limits, and budget for the requests you send.

Three different things get called "failover"

Before testing, separate what the vendor might actually be doing. The behaviours look identical in a marketing page and completely different in a response body:

What happens on a 429 What it is What you experience
Retry the same upstream after a delay Retry / backoff Slow response, then success or the same 429
Switch to a different provider for the same model Failover May restore service; verify model version and compatibility
Switch to a different, cheaper model Fallback Success, different output quality, different price

These are the distinctions used in this article; vendors may name them differently. An SDK retry can eventually succeed without switching upstream. A model fallback can improve availability while changing the model, so its compatibility and quality need separate validation.

If a relay cannot explain which behaviour it implements, treat that behaviour as unknown rather than inferring it from the marketing name.

What a reported issue can tell you

LiteLLM #24152 reported a specific case in v1.81.9-stable: a per-key, per-model rate limit was enforced before the router, so the configured fallback was not reached. It is a closed historical report, not evidence that every 429 or the current release behaves that way.

The useful question is which layer rejected the request. A customer-key quota limit, an upstream quota limit and an empty candidate pool can all look like a failed call, but require different fixes. Do not assume that a relay should route around a limit intentionally applied to your account.

The test

You need: an API key, curl or a five-line script, and a model name that maps to a group with more than one upstream behind it.

Step 1 — Baseline

Send 20 short requests, sequentially, and record for each: HTTP status, latency, usage, and any model identifier in the body.

for i in $(seq 1 20); do
  curl -s -o /tmp/r.json -w "%{http_code} %{time_total}\n" \
    -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d '{"model":"your-group","messages":[{"role":"user","content":"say ok"}],"max_tokens":5}' \
    https://your-relay.example/v1/chat/completions
done

Keep each response body and the timing output; the example overwrites /tmp/r.json, so change the filename per request if you need the full sample. Twenty requests give a rough baseline, not a stable long-term p95. Record timeouts and streaming interruptions too.

Step 2 — Arrange a controlled failure

Prefer an isolated or explicitly authorised test: upstream A returns an agreed pre-generation rejection, upstream B remains healthy, and the operator can show the attempts belonging to that same request. Do not exhaust a shared provider’s quota just to provoke a fault.

An error caused by your own customer-key allowance tests admission limits, not necessarily upstream failover. If you cannot induce a controlled failure or obtain a trace, keep the conclusion at “consistent with failover” or “unverified”.

Step 3 — Read the error body, not the status code

This is where the test actually happens. When the failure arrives, look at the JSON:

{"error":{"message":"upstream idle timeout","type":"upstream_error"}}

These are clues about the final error, not proof of how many upstreams were attempted:

  • a provider name (Together, DeepInfra, Anthropic, Azure, a region, an account id)
  • rate limit / quota exceeded wording that names a third party
  • upstream idle timeout, upstream_error, provider not found, no available channel
  • a vendor-specific error code that your own request could not have produced

A successful response shows that the request completed; it does not, by itself, prove a switch. The first candidate may have succeeded. Conversely, an upstream error may appear after several candidates were tried and all failed. Correlate the final response with a request trace. For streaming calls, check completion rather than only the initial HTTP 200.

Step 4 — Compare timing with the trace

Another attempt can add time for rejection, candidate selection and a new request. Queueing, model load, caches and transport can also change latency; there is no universal 1–3 second signature.

  • A fast 429 may be a customer limit, an immediate upstream rejection or a cooling-down decision. It does not prove that no retry was attempted.
  • A slower successful response is consistent with another attempt, but does not identify one.
  • Stronger evidence is an accepted test request whose trace shows A rejected, B attempted, and a completed final result under the same request ID.

Reading the result

Observation What you can conclude
Upstream or quota wording in the final error Classify the final failure; the attempt count is still unknown.
Repeated fast 429s Inspect the limiting layer and cooling policy. Speed alone is not proof.
Success after an induced A failure, with a trace showing B Evidence that failover worked in that test.
A different model identifier May indicate model fallback; check whether the identifier is an alias.
200, unchanged model field, similar latency Does not distinguish a switch from a healthy first candidate.

Output style is not a reliable provider fingerprint. Sampling, model versions and serving settings can change it. Likewise, a response model field may be a group alias rather than the actual member.

What this test cannot prove

Be honest about the limits, because the marketing side won't be:

  • It cannot prove the group has more than one upstream. A single-upstream group that never fails passes every test above.
  • It cannot measure failover under real load. A relay that switches correctly at 1 request per second can still collapse at 50 concurrent requests if its candidate list is short.
  • It cannot distinguish "switched" from "never failed". If the primary was healthy the whole time, both look like a 200. Forcing a failure is the only way to tell them apart, and sometimes you cannot force one from outside.
  • It says nothing about data handling. Routing a request to a second provider means a second provider saw the prompt.

That last point is why an operator who publishes which upstreams sit behind a group, and how failures are classified, is worth more than one who publishes a 100% uptime badge.

The short version

  1. Take a small authorised baseline and keep response, timing and completion data.
  2. Arrange a controlled failure with a healthy backup.
  3. Use the error body to classify the failure, not to infer the entire attempt history.
  4. Confirm the switch with a correlated request trace. Treat timing and model fields as clues.

If a trace is unavailable, report that limit rather than claiming that a black-box success proves failover or that a final error proves no switch.


We run XCoffee, a task-grouped model API with retry on eligible rejections before generation. Its public model fields identify groups, so they do not reveal which member answered. This article explains how to assess evidence; it does not claim that a response field alone verifies XCoffee’s internal attempts. Related: timeout diagnosis and retry vs fallback vs failover.