An agent that retries every exception is not resilient. It is a loop that can send the same email twice or keep hitting a service that has already said no. Retry logic comes down to three decisions for each failure: can it recover, how long should the agent wait, and is another attempt safe to make?
This guide covers failure classification, backoff, and side effects that must not run twice. It reports no measured improvements, and the numbers in the code are starting points that you should test with simulated failures. For visibility into what your retries are doing, see the companion guide on monitoring an agent.
Sort failures before choosing a response
Every failure fits one of four groups, and the group decides the response.
Transient. The service or network had a temporary problem. Anthropic's API error reference says to retry a 500 internal error with exponential backoff, describes a 529 as a temporary overload, and says its official SDKs automatically retry connection errors, rate limits, and 5xx errors twice by default, honoring the retry-after header when present.
Permanent. The request itself is wrong. Google's Calendar error guide says a 400 is a permanent error, so do not retry: read the message and change the request. Sending the same bad request again only adds load.
Needs a person. The agent cannot fix this alone. A revoked login, a billing problem, or an administrator policy will not clear with waiting. Google's Gmail error guide tells clients to refresh an expired token once and, if that fails, send the user through authorization again. For a domainPolicy error it says to tell the user to contact their domain administrator.
Ambiguous. The call timed out or the connection dropped, and you do not know whether the other side acted. For a read, this is just transient. For a write, it is the dangerous case, covered below.
Status numbers alone mislead. A 429 is usually transient, but Anthropic's reference describes a 429 caused by reaching a usage tier's monthly spend cap that has no retry-after header and keeps failing until access resumes. A Gmail 403 can mean a rate limit, which merits backoff, or a domain policy, which does not. Read the structured reason field when the API provides one. Vendor pages also differ from generic rules in places: the Calendar guide suggests backoff for a 404, so check the page for the specific API you call.
import random
import time
TRANSIENT_STATUS = {429, 500, 502, 503, 504, 529}
RETRYABLE_403_REASONS = {"rateLimitExceeded", "userRateLimitExceeded"}
def classify(status, reason=None, may_write=False, replay_safe=False,
requires_attention=False):
"""requires_attention is set by a provider-specific error adapter."""
if requires_attention:
return "needs_person"
if status in (401, 402) or (status == 403 and reason not in RETRYABLE_403_REASONS):
return "needs_person"
transient = (status is None or status in TRANSIENT_STATUS
or (status == 403 and reason in RETRYABLE_403_REASONS))
if transient:
# A server error can follow a committed write, not only a timeout.
if may_write and not replay_safe:
return "ambiguous"
return "transient"
return "permanent"
def call_with_retries(fn, may_write=False, replay_safe=False,
max_attempts=4, base=1.0, cap=32.0, deadline=120.0):
if max_attempts < 1:
raise ValueError("max_attempts must be positive")
start = time.monotonic()
for attempt in range(1, max_attempts + 1):
try:
return fn()
except ApiError as err: # supplied by your client adapter
kind = classify(err.status, err.reason, may_write, replay_safe,
err.requires_attention)
if kind != "transient" or attempt == max_attempts:
raise
delay = min(cap, base * 2 ** (attempt - 1) + random.uniform(0, 1))
if err.retry_after is not None: # adapter parses seconds or HTTP-date
delay = max(delay, err.retry_after)
if time.monotonic() - start + delay > deadline:
raise
time.sleep(delay)
ApiError is an adapter you supply, with the attributes used above. Set may_write from the operation type, not from whether you know it succeeded. A timeout or server error on a write is ambiguous unless the service guarantees safe replay with the same key. The adapter must mark spend caps and other account conditions as requires_attention; status alone is insufficient. It must also parse Retry-After into seconds, whether the header contains seconds or an HTTP date. Give each request its own timeout. The deadline in this sketch limits waits, not a blocked call.
Bound the retries and spread them out
Gmail's quota page describes truncated exponential backoff: wait about 2 to the power n seconds plus a random amount of up to a second, capped at a maximum backoff that is typically 32 or 64 seconds. The random part matters. Amazon's Builders' Library article on timeouts, retries, and backoff with jitter points out that traffic often arrives in bursts, including bursts from a periodic cron job, and that retries are less effective when all clients retry at the same time. Jitter, a random delay before a request, spreads them out. Scheduled agents are exactly this kind of client, so add jitter to the start time of a job as well as to each retry.
The same article describes a second trap: retries multiply across layers. In its example, three retries at each of five layers can turn one failing call into 243 calls against the database at the bottom. Your agent has layers too. If the SDK already retries twice, each attempt of your wrapper can issue up to three calls. Four wrapper attempts then means up to twelve. If the scheduler also reruns a failed job, multiply again. Pick one layer to own retries and set the others to zero, or at least know the product of all of them.
Timeouts belong in the same design. The Amazon article advises a timeout on every remote call, noting that too high a value wastes resources and too low a value causes needless retries. Model calls are slow, so Anthropic's error reference recommends streaming for long requests. It also says an error can occur after a streaming response has already returned a 200. Treat a complete stream as the success signal, not the first status line, and never treat partial output as a finished result.
Handle ambiguous writes with keys and lookups
RFC 9110 defines a request method as idempotent when repeating an identical request has the same intended effect as sending it once. It lists PUT, DELETE, and the safe methods as idempotent, and it says a client should not automatically retry a non-idempotent request unless it knows the request is effectively idempotent or can detect that the original was never applied. It adds that a proxy must not automatically retry non-idempotent requests. A POST that creates something is the typical hazard.
There are two ways to make a write safe to repeat:
- Send an idempotency key. Stripe's idempotent requests documentation describes the pattern: the client supplies a unique key, the server saves the result of the first request made with it, and a retry with the same key returns the saved response. Stripe also compares the parameters and returns an error if a retry changes them, advises against using sensitive data such as email addresses as keys, and notes that keys can be removed after at least 24 hours in its v1 API. Other services differ, and many offer no keys at all.
- Look the result up before retrying. If you can search the destination by something you chose, check it after a timeout. Google's Calendar guide documents a 409 duplicate response when an event ID already exists. If you supply your own event ID when creating an event, a retry that returns 409 tells you an event with that ID exists. Fetch it and compare it with the intended event before declaring success; an ID collision is not proof that your first attempt succeeded. Check the events reference for the ID format rules before relying on this.
The diagram shows the pattern with a durable intent record. Before the write, save a row that holds the key, the intended action, and the status attempting. After a timeout, look up the result by that key. If it exists and matches the intended action, mark the row succeeded. A missing result may reflect stale reads. Retry only when the service guarantees safe replay with the same key, or you have authoritative evidence that the operation was never applied. Otherwise mark it unknown and send it to a person. For anything sent to another human, such as an email, the cleanest answer is to skip automatic sends and queue a draft, which is why guides like the Gmail read-only design never send.
Persist state so a restart cannot lose or repeat work. A job table with the states queued, attempting, retry_wait, succeeded, failed_permanent, needs_person, and unknown is enough, along with a next_attempt_at time so a restarted process resumes the wait instead of skipping it.
Model output failures are a different kind of retry
A model returning malformed JSON is not a network blip. Another call may succeed, but only a bounded number of tries makes sense. One repair attempt that includes the validation error is reasonable. After that, send the item to the review queue with the error attached. Output cut off by a length limit also needs a change of input or settings, not a blind resend. Do not retry a refusal as if it were an outage.
Resume from the failed step. If a multi-step run has already created a record or sent a message, rerunning from the top repeats those effects. Checkpoint after each step with a side effect and restart from the checkpoint.
Record enough to diagnose, and test with injected failures
For each failure, log the error class, status, reason, attempt number, and the provider's request ID. Anthropic's error responses include a request_id for exactly this purpose. Do not log secrets or message bodies.
Then test the logic with a fake client instead of waiting for real outages. Make it return a 503 twice and then succeed, return a 400, time out after a write, and return a 401 that cannot be refreshed. Assert on the number of calls and the final state of each job. Record those outcomes using the monitoring pattern.
Limits of this approach
Classification tables age, because APIs add error types. Backoff settings that suit one service can be wrong for another. A lookup after a timeout can itself fail or return stale data. None of this removes the need for a person to look at items marked unknown or needs_person. If you are building your first worker, begin with the first-agent guide and add these rules one write action at a time.
Sources
- Anthropic, Claude API errors: status codes, SDK retry defaults, spend-cap 429 behavior, request IDs, and mid-stream errors.
- Google, Gmail API error handling and usage limits: token refresh guidance, 403 reasons, and the truncated exponential backoff formula.
- Google, Calendar API errors: permanent 400s, 409 duplicate IDs, and 412 precondition failures.
- Amazon Builders' Library, Timeouts, retries, and backoff with jitter: timeouts, jitter, cron bursts, and multiplied retries across layers.
- IETF, RFC 9110 HTTP Semantics: idempotent methods and when automatic retry is allowed.
- Stripe, Idempotent requests: idempotency key behavior.


