LLM API retries and fallback: avoid applying timeout budgets twice

Symptoms

A logical request exceeds its intended timeout because a retry or secondary provider receives a new full allowance. A second retry layer inside an SDK can multiply attempts again. The reviewed application dependency pins are openai==2.46.0, anthropic==0.117.0 and google-genai==2.8.0; these identify the reviewed revision, not the newest available releases.

Cause

The reviewed factory creates one monotonic deadline and calls remaining_provider_timeout before each leg. Its OpenAI and Anthropic constructors disable automatic retries with max_retries=0; the Gemini constructor limits SDK retry attempts to one. Gemini’s per-request timeout is converted from seconds to milliseconds. The official OpenAI Python reference separately documents configurable SDK retries. The application’s retry policy and fixed ceilings are application decisions, not provider limits.

Safe resolution

The reduced function uses one clock/deadline and an injected provider boundary. Configure SDK retry ownership outside this function; a provider callable must honor the timeout passed to it and must not secretly reset the logical deadline.

def call_with_budget(providers, *, budget_s, clock):
    if budget_s <= 0:
        raise ValueError("Budget must be positive")
    deadline = clock() + budget_s
    last_error = None
    for send in providers:
        remaining = deadline - clock()
        if remaining <= 0:
            raise TimeoutError("Logical LLM deadline exceeded")
        try:
            result = send(timeout_s=remaining)
        except TimeoutError as exc:
            last_error = exc
            continue
        if clock() > deadline:
            raise TimeoutError("Logical LLM deadline exceeded")
        return result
    if last_error is not None:
        raise last_error
    raise RuntimeError("No provider available")

Verification

Synthetic Python examples were checked on 2026-10-16; the original application tests and live provider APIs were not run.

Use a fake monotonic clock rather than sleeping. With a budget of ten seconds, let the primary consume six seconds and time out; assert the secondary receives only four seconds. If the primary exhausts the budget, assert secondary calls remain zero. Reject a result returned after the deadline. Preserve a successful first response without starting fallback. Inspect the real adapter separately for max_retries=0, Gemini attempts=1 and millisecond conversion.

The checked-in tests extract this published Python block and exercise the same synthetic boundary cases. Re-run with npm test.

Limitations

Checking a local deadline does not prove a blocking transport is immediately cancelled; a timed-out generation may still have reached a provider. Retry side effects and request authority/cancellation need their own boundaries. This example retries only TimeoutError, not arbitrary permission failures or malformed output, and does not reproduce the historical adapter’s source-specific authentication retry policy. Backoff must also consume the same budget and release concurrency slots. See the retry guide and OpenAI profile.

Implementation evidence was reviewed at pinned GetBeta revision 890c2531b76c. Source-repository links may require access; the public code above is a reduced adaptation without credentials or customer data. Search expressions are candidates, with no search-volume, indexing or ranking claim.