Your Relay's Status Page Says 100% - Here's What It Is Actually Measuring
Why a green status page may not describe your model request: check monitoring scope, completed outcomes, routing evidence, sample size and measurement limits.
Your Relay's Status Page Says 100% - Here's What It Is Actually Measuring
There is a specific kind of failure that only happens to relay users, and it has a signature: your requests are failing, the errors are unmistakably upstream, and the provider's status page says everything is fine.
One possible reason is a difference in what is measured. First inspect the monitor’s scope instead of assuming that a green badge covers your entire request path.
What a status page actually measures
A status page may measure only the relay’s HTTP surface: did the gateway accept the connection, did it return a response, how long did that take. That is a real measurement of something. It is just not a measurement of your request.
If the monitor stops there, it misses the second half of your request: after the relay accepts it, the relay has to pick an upstream, authenticate against it, and wait for a model to answer. Every layer in between can fail while the badge stays green.
| Layer | Who watches it | Typical failure |
|---|---|---|
| Relay frontend / API edge | Gateway reachability monitor | Connection, edge or application failure |
| Upstream selection (which provider) | Routing telemetry, if instrumented | Empty candidate list, all providers cooling down |
| Upstream authentication | Sometimes | Key revoked, quota exhausted, 429 |
| Model serving | The upstream's status page | Timeout, overload, 502 |
Providers can show 100% for the fourth row while your request dies in the second. And even when the provider is up, the relay's own routing can send you to the wrong one: a group that picks the cheapest upstream, or the one with the most free quota left, is picking on a metric that has nothing to do with whether the model is healthy right now.
A sequence of failed requests alongside a green status page establishes a mismatch between your observations and that page’s scope. It does not, by itself, establish what the operator monitors. Ask whether availability is based on HTTP reachability, completed model calls or another definition, and which regions and groups are sampled.
The three numbers worth asking for
If a status page measures the wrong layer, the fix is to measure the right one. You do not need a monitoring platform for this. You need three facts about each request:
1. Which upstream answered it
Ask which serving route handled the request, rather than relying on a model label. A stable anonymised route ID and a request ID can support diagnosis without publishing account names, credentials or sensitive infrastructure details.
2. Whether a fallback happened
Not whether an error was returned - whether the request was retried elsewhere and succeeded. Another attempt can add latency, but timing alone does not prove one. Correlate a switch indicator with the attempts belonging to the same request.
3. Whether the fallback worked
A relay can switch and still fail, if every candidate is degraded. So the interesting number is not "we have N providers" but "what fraction of requests ended up served by provider X, and how often did we have to leave provider X."
How to get them without changing your code
Some gateways expose routing metadata in response headers. The following names are illustrative, not a standard or a claim about headers currently returned by XCoffee:
x-upstream-provider: together
x-fallback: true
x-upstream-attempts: 2
x-upstream-latency-ms: 1840
The names differ per vendor; the shape does not. If a request comes back with a provider name in a header, you can log it, count it, and see the distribution:
curl -sS -D - -o /dev/null \
-H "Authorization: Bearer ***" \
-H "Content-Type: application/json" \
-d '{"model":"your-group","messages":[{"role":"user","content":"say ok"}],"max_tokens":5}' \
https://your-relay.example/v1/chat/completions \
| grep -iE 'x-(upstream|provider|fallback|attempt)'
An authorised sample can show your observed distribution if the route IDs are accurate and completion is recorded. Record the window and sample size. For SSE, an initial HTTP 200 is not the same as a completed generation.
If your relay exposes nothing
You can still measure final request outcomes and classify error clues. The following wording may point toward an upstream or quota problem, but it cannot identify a provider reliably or establish the attempt history:
- a provider name (
Together,DeepInfra,Anthropic,Azure, a region, an account id) - third-party quota wording (
rate limit,quota exceeded,insufficient credits) - relay-specific upstream errors (
upstream idle timeout,upstream_error,no available channel)
Log HTTP status, latency and error-pattern matches. That gives a request-level error-hint rate, not a per-provider success rate. The simple example below also needs separate timeout and streaming-completion handling for a full test:
import re, time, requests
LEAK = re.compile(r"together|deepinfra|rate limit|quota|upstream|no available channel", re.I)
def call(relay, key, group):
t0 = time.perf_counter()
r = requests.post(f"{relay}/v1/chat/completions",
headers={"Authorization": f"Bearer {key}"},
json={"model": group, "messages": [{"role": "user", "content": "say ok"}],
"max_tokens": 5}, timeout=60)
ms = (time.perf_counter() - t0) * 1000
body = r.text
upstream = LEAK.search(body)
return {"status": r.status_code, "ms": round(ms),
"error_hint": upstream.group(0) if upstream else None}
Take a small authorised sample and report the final error rate with its sample size and window. An error-pattern match is a hint, not proof that failover was absent or that one particular provider failed. Do not claim a per-provider rate without reliable attribution.
The same question, asked properly
When you raise this with a relay, ask three specific questions instead of "what is your uptime":
- Which upstream served my last request? (Do they answer, or do they refuse?)
- What is my measured fallback rate - the share of my requests that were retried on a different upstream because the first one failed?
- What is the per-provider success rate on my traffic, as opposed to your fleet average?
An operator’s ability to supply correlated, appropriately redacted evidence is more useful than a badge alone. If that evidence is unavailable, record the limitation rather than inventing an attribution.
What this does not prove
- Provider attribution is not health. Knowing the request went to provider X tells you where it went, not whether that provider was healthy or merely reachable.
- A fleet average is not your rate. For example, an aggregate 99.9% can coexist with a 5% error rate in one group or time window; these are illustrative figures, not measurements of XCoffee.
- Header visibility does not mean failover works. A relay can proudly report
x-fallback: truewhile failing to switch anything. Still run the black-box failover test once, so the header is corroborated rather than trusted. - Fallback is not free. If a request is retried on a second provider, a second provider saw your prompt. That is a data-handling fact worth knowing before you rely on failover.
The short version
- Check what the status page actually measures. Gateway reachability alone is not proof that your model request completed.
- Ask for three per-request facts: which upstream answered, whether a fallback happened, and the per-provider success rate on your traffic.
- If attribution is unavailable, log outcomes, latency and error hints. Report request-level results and the visibility limit, not a per-provider rate.
One relay claiming 100% uptime while your requests fail tells you nothing about the relay. A relay that tells you which upstream answered, how often it switched, and how often the switch worked - tells you everything.
We run XCoffee, a task-grouped model API. This article explains useful evidence and measurement limits; it does not claim that XCoffee currently exposes the illustrative routing headers or a public per-provider status dashboard. Related: how to assess failover, timeout diagnosis, and retry vs fallback vs failover.