Retry Logic

You are a resilience engineer implementing retry logic with exponential backoff for network requests.

What to check first

• Verify your HTTP client library supports timeouts (e.g., requests, axios, fetch)
• Confirm maximum retry attempts and initial delay values match your SLA requirements
• Check if your service has rate-limiting headers (Retry-After) that should override backoff calculations

Steps

1. Define retry configuration with initial_delay (milliseconds), max_retries, backoff_multiplier (typically 2), and max_delay cap
2. Identify which HTTP status codes trigger retries: typically 429 (rate limit), 503 (unavailable), 504 (gateway timeout), and timeout errors
3. Implement jitter by adding random variance to each delay to prevent thundering herd when multiple clients retry simultaneously
4. On each failed request, calculate next delay as min(initial_delay * (multiplier ^ attempt_number) + jitter, max_delay)
5. Check for Retry-After header in 429/503 responses and use that value if present instead of calculated backoff
6. Log retry attempts with attempt number, delay duration, and reason to enable debugging of transient failures
7. For idempotent requests only (GET, PUT with idempotency keys), retry safely; avoid retrying POST without idempotency guarantees
8. Set total timeout across all retries to prevent indefinite blocking when service is down

Code

import time
import random
import logging
from typing import Callable, TypeVar, Optional
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry as Urllib3Retry

logger = logging.getLogger(__name__)

T = TypeVar('T')

class ExponentialBackoffRetry:
    def __init__(
        self,
        initial_delay: float = 1.0,
        max_retries: int = 5,
        backoff_multiplier: float = 2.0,
        max_delay: float = 60.0,
        jitter: bool = True
    ):
        self.initial_delay = initial_delay
        self.max_retries = max_retries
        self.backoff_multiplier = backoff_multiplier
        self.max_delay = max_delay
        self.jitter = jitter

    def calculate_delay(self, attempt: int) -> float:
        """Calculate backoff delay with optional jitter."""
        delay = self.initial_delay * (self.backoff_multiplier ** attempt)
        delay = min(delay, self.max_delay)
        
        if self.jitter:
            delay *= (0.5 + random.random())
        
        return delay

    def retry(
        self,
        func: Callable[..., T],
        *args,
        **kwargs
    ) -> T:
        """Execute function with exponential backoff retry logic."""
        last_exception = None

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change