XCoffee
Back to blog

80% of Your Requests Time Out and Failover Never Fires - a Diagnostic Order

Nov 11, 2026XCoffee Team

When most of your LLM requests time out but failover supposedly handles it, find the real failing layer: client, network, relay, upstream - and the three numbers to collect at each.

80% of Your Requests Time Out and Failover Never Fires — a Diagnostic Order

Consider a report from an agent loop: roughly 80% of requests either time out or fail entirely, and the operator tried every usual trick — pin fewer providers, only pin fast providers, raise the client timeout — and nothing moved. When a request that “should fail over” instead dies, locate the failing deadline or path before assuming that the fallback implementation is the cause. The caller may have a timeout rather than an error body to inspect.

This is a triage order. It does not tell you what to buy; it tells you which layer is broken and who to ask. The signatures below are diagnostic clues, not exclusive proofs. Find the likely failing layer before changing the next one.

Rule the whole thing by one question first

Before touching anything, separate timeouts from error codes. They are different failures:

  • A timeout means the caller did not receive the expected result before a deadline. The server may still have accepted or processed the request. A connection reset is a separate transport failure; neither establishes whether replay is safe.
  • A 429 / 400 / 401 / 5xx means an answer came back, even if it is the wrong one.

If most failures are timeouts, first locate the deadline: client, edge, relay, connection or stream-idle timer. Check whether the relay’s total budget leaves enough time for another attempt before the client gives up. A timeout can be eligible or ineligible for retry depending on policy and whether work may already have started. It does not automatically mean that fallback code is broken—or that the problem is only configuration.

The order

Layer 1 — Client

Collect, for a 30-minute window, three numbers: p50 latency, p95 latency, and the share of timeouts vs. error codes.

# per request, append to a log: t_start, t_end, status, body_snippet

If measured p95 is close to the client timeout, investigate whether the deadline truncates your observations. A 15 s timeout with a 14 s p95 leaves little margin, but cannot explain an 80% failure rate by itself. Include failed and timed-out calls in the analysis before changing the limit.

Layer 2 — Network

Between the client and the relay:

  • Are timeouts concentrated in one region or one egress? If a specific office VPN or a specific cloud region sees all the failures, it is a path problem, not a service problem.
  • Do timeouts cluster at a specific time of day? That can point to load, quota windows or maintenance, but needs corroboration.
  • Is the connection ever established at all? Record DNS, TLS, connection and first-byte timings where possible. A failure before headers or after the first byte can originate at several layers, including a proxy or the upstream.

A possible clue to a network-layer problem: connection reset or no response, no status code, and it varies by location or time.

Layer 3 — Relay

This is where "failover" lives. Ask the operator these five questions, in this order, because each one isolates a different breakage:

  1. On a 429 from an upstream, do you retry the same upstream, switch to a different provider, or switch to a different model? If the answer is "we try another one", which one, and is that behaviour logged?
  2. How many candidates sit behind each group? A group with one upstream has no failover to offer; it can only retry or time out.
  3. What are the per-attempt and total request deadlines, and what happens to a failed candidate? Check that the total retry budget fits inside the client’s deadline; simply making one timeout larger does not establish a working failover path.
  4. When an upstream errors, does the error body reach the client, or is it translated? A provider name identifies an error source, but does not tell you whether other candidates were tried first.
  5. Is there a cool-down after an upstream failure, and how long? Ask whether another candidate is tried in the same request, what happens when every candidate is cooling, and how a candidate becomes eligible again.

A possible clue to a relay-layer problem: a status code comes back (429, 502, 503), sometimes you can read an upstream's name in it, and the behaviour is identical no matter how many times you retry.

Layer 4 — Upstream

If traces confirm that the relay is switching but no attempt succeeds, investigate the candidates and their shared dependencies. That is the "80% of requests time out" case taken to its logical end: not one bad provider, but the whole pool under load.

Check:

  • Is the specific model you are calling the one that is scarce? Check that model’s actual capacity and account limits rather than inferring scarcity from its price.
  • Did the failure start at a specific moment? A model deprecation, a provider's capacity change, or a price/cap update often shows up as a step-change in your error rate.
  • Are you on a free or shared tier? Shared-pool capacity means your timeout is decided by everyone else's load, and no amount of client-side pinning fixes it.

A possible clue to a upstream-layer problem: the error rate jumped at a specific point in time and stays elevated; pinning a different provider made it worse or made no difference.

Why limiting the provider list can make it worse

The most common misdiagnosis is to respond to 80% timeouts by narrowing the provider list — "only fast providers", "only provider X". On a shared pool, fewer candidates can reduce the capacity available to absorb a burst, so narrowing the list may increase timeouts. Use per-route measurements rather than assuming that this explains a particular incident. Narrowing helps when one provider is reliably broken; it hurts when the whole pool is the constraint. Read which case you are in before cutting the list.

The three numbers that end the debate

You do not need a full observability stack. Per request, keep exactly these and a timestamp:

Number What it rules out
p50 and p95 latency A too-low client timeout (layer 1)
share that is timeout vs. status code Whether any answer is coming back at all
error rate over time, with a jump detector Whether a provider/model changed (layer 4)

These figures help choose the next investigation; none identifies a cause alone. Separate successful calls, failed calls and deadline-truncated samples. A step change can follow an upstream, relay, client or network change. A provider name in the final body still does not reveal the complete attempt history.

What to do per layer

  • Layer 1: check the end-to-end deadline budget and cancellation behaviour; adjust a too-short client timeout with a bounded margin, then re-measure.
  • Layer 2: pin the failing requests to a region and time; escalate as a path issue, not a service issue.
  • Layer 3: send the operator the five questions above plus one forced-failure trace. If no trace is available, record that the routing behaviour remains unverified; missing visibility does not establish pool size.
  • Layer 4: stop pinning the scarcest model for bulk work; move batch jobs to a tier with real capacity, and keep the interactive path on the fast one.

None of this is a recommendation to switch products. It is a way to know which layer owns the fix before you spend money or rebuild the retry logic on the wrong layer.


XCoffee runs task-grouped model APIs with in-group retry before generation. The diagnostic order above is the one we use when a customer reports a high error rate: which layer is failing first decides the fix. See also how to verify a relay's failover is real and retry vs fallback vs failover.