Can I set max_retries for requests.request?

python

retry-strategy

requests-library

resilience

byNikita Barsukov·Sep 19, 2024

To apply max_retries in requests, configure a retry strategy and bind it to a requests session. Here's your "cheat sheet":

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Funny but true: Error 5xx? More like error high-five, let's try again!
retries = Retry(total=5, backoff_factor=0.1, status_forcelist=[500, 502, 503, 504])

# Session initiation level: "Pro".
session = Session()
session.mount('http://', HTTPAdapter(max_retries=retries))
session.mount('https://', HTTPAdapter(max_retries=retries))

response = session.get('https://example.com')

Don't forget, HTTPAdapter needs to be mounted for both http and https protocols. Dial session.get to make your resilient requests.

Demystifying retry parameters

A perfect retry strategy hinges on understanding each Retry parameter. Here's what they do:

total: Overall retries allowed before throwing in the towel.
connect: For when you just can't connect to the Wi-Fi at the café.
read: For those pesky "unable to read" issues.
redirect: For agents who always accept redirection in their mission.
status_forcelist: A list of status codes that scream "yes, please try again!"
backoff_factor: The 'cooling off' period between each retry attempt.
allowed_methods: HTTP methods that you permit for retrying.

Configure these to suit your app's resilience level and failure handling.

Pitfalls to avoid when setting retries

Globally setting DEFAULT_RETRIES is as messy as a sloth painting a portrait. Don't! Instead, master the art of session-based retries.

One more thing: monkey-patching is a no-no. It's an unstable and messy way to set your max_retries.

Keeping track of the updates

Remember, requests is an evolving library. So, keep an eye on those change logs. Sudden features like raise_on_status can appear/disappear, altering your retry strategy's course.

Additional insights

Advanced retry mechanics

Consider custom retry logic for systems with intricate needs. Circuit Breakers can prevent service overwhelm, and jitter can avoid synchronized retries.

Negotiating with rate limits and backoff

When faced with rate limits, dynamically modify backoff_factor or devise your own backoff mechanism to respect the limits while sustaining the process.

Persistent strength

Utilize session objects for persistence, avoiding the need for repeated transmission of headers or cookies.

Thinking about performance

Contemplate that retry attempts can add latency. Balancing this performance impact requires careful monitoring of response times against the advantages of strengthened resilience.

explain-codes / Python / Can I set max_retries for requests.request?

Linked

Correct way to try/except using Python requests module?



Java.net.socketexception: Connection reset



Error Code: 2013. Lost connection to MySQL server during query



How to handle errors with boto3?



Why is "except: pass" a bad programming practice?

