What is the fastest way to send 100,000 HTTP requests in Python?

python

async-functions

concurrent-programming

http-clients

byAnton Shumikhin·Jan 13, 2025

The quickest way to dispatch 100,000 HTTP requests in Python is using asyncio coupled with aiohttp for asynchronous operations. It enables sending requests parallelly without awaiting each to finish. Let's examine a concise code that exhibits how this is done:

import asyncio
import aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [session.get('http://example.com') for _ in range(100000)]
        await asyncio.gather(*tasks)

asyncio.run(main())

The above code snippet demonstrates how to generate and dispatch all requests concurrently, leveraging the event loop for managing network I/O operations.

Minimizing latency and handling exceptions

Reducing latency and handling exceptions efficiently are the keys to optimizing our async functions:

Utilize a ClientSession: Creating a session to reuse connections across multiple requests using aiohttp.ClientSession(), effectively reducing overhead.
Handle exceptions: Incorporate try-except blocks within your coroutine to gracefully handle ClientConnectorError and other potential mishaps during high concurrency.
Optimize with HEAD requests: Resort to session.head('http://example.com') if you're merely inspecting the existence of a URL, as HEAD requests fetch only headers and are considerably faster.
Connection limits: Set a ceiling to the number of parallel connections to prevent flooding the local client or remote server.

Other techniques and considerations

Traditional threading and multiprocessing

Despite the proficiency of asyncio and aiohttp, scenarios may arise where classical threading or multiprocessing methods are required:

Threading: Ideal when dealing with I/O-bound tasks, but the Global Interpreter Lock (GIL) in CPython could limit its performance.
Multiprocessing: Applicable when managing CPU-bound tasks or escaping the GIL, keeping in mind the increased use of memory and potential overhead from inter-process communication.

Modern HTTP clients

HTTPX and grequests are newfound libraries offering async capabilities:

HTTPX: This library supports HTTP/2 and asynchronous requests, with a familiar API for those accustomed to requests.
grequests: Essentially requests on steroids, built on gevent, it allows handling asynchronous HTTP requests.

Utilizing concurrent.futures for task handling

concurrent.futures.ThreadPoolExecutor or concurrent.futures.ProcessPoolExecutor are highly efficient for maintaining a pool of threads or processes:

from concurrent.futures import ThreadPoolExecutor, as_completed

def send_request(url):
    # "You don't need a silver bullet, just a good scapegoat"
    pass

urls = ['http://example.com' for _ in range(100000)]
with ThreadPoolExecutor(max_workers=10) as executor:
    future_to_url = {executor.submit(send_request, url): url for url in urls}
    for future in as_completed(future_to_url):
        url = future_to_url[future]
        try:
            # "Mornings are for coffee and contemplation"
            data = future.result()
        except Exception as exc:
            print(f'{url} generated an exception: {exc}')

Alternative high concurrency approaches: Tornado and Twisted

For managing high concurrency without falling back to asyncio, consider platforms like Tornado and Twisted:

Tornado: A Python web framework and asynchronous networking library, utilizing non-blocking network I/O.
Twisted: A proficient event-driven networking engine that operates asynchronously. Ideal for situations involving long-lived network connections.

Performance measurement and optimization

To benchmark and improve your solution:

Time measurement: Resort to Python's time module to track the commencement and termination of the process to measure total execution time.
Connection and timeout tuning: Adjust the number of concurrent connections and set sensible timeouts to maximize performance without overwhelming the network stack.

explain-codes / Python / What is the fastest way to send 100,000 HTTP requests in Python?

Linked

How do I use threading in Python?



What's the difference between Jetty and Netty?



How to check internet access on Android? InetAddress never times out



Sending "User-agent" using Requests library in Python



Timeout for python requests.get entire response

