How can I see the entire HTTP request that's being sent by my Python application?

python

logging

requests

debugging

byAlex Kataev·Oct 19, 2024

Activate HTTP tracing in Python using this simple method: http.client.HTTPConnection.debuglevel = 1. This reveals the raw HTTP request data in your console. The code snippet for a simple requests session is:

import http.client
http.client.HTTPConnection.debuglevel = 1  # Crank up the verbosity
import requests

response = requests.get("http://example.com")  # Get me some interweb bytes

You'll get a gaze at your headers, body, and more right in the console chimney. Great for debug-level chimney sweeps. Mind your production environment, though!

Deep Dive: What You'll See

Setting debuglevel = 1 in the http.client.HTTPConnection gives you a peek into your HTTP parlour. This unlock includes:

The request line itself (like GET / HTTP/1.1)
All those neat request headers (like Host: example.com)
The body of the request when it's chatty (like with POST requests)

It also constructs a fetching tableau of the server's response, including status codes, headers, and body.

Exploring Complex Curves

When your HTTP request decides to take a detour, or redirects, check r.request.history for a play-by-play of the relay.

Authentication cases? Naturally, r.request.auth spills the beans about the credentials used.

Consider enchancing your requests with hooks to make your code Sherlock-keen at intercepting lifecycle events:

def logging_hook(response, *args, **kwargs):
    print(response.request.headers)  # Spill the headers, love
    print(response.request.body)  # Body, tell all

hooks = {"response": logging_hook}  # a hook for our inspector
response = requests.get("http://example.com", hooks=hooks)  # Magic!

The output? Self-updating, blow-by-blow details of each request and response. Neat, huh?

Tooling It Up Deeper

Not a fan of code? Use HTTP Toolkit for a graphical peek into your HTTP traffic without coding.

If command-line's your jam, unleash netcat (like a chatroom for your data) with ✨ nc -l 1234 ✨.

For a more holistic approach, mitmproxy lets you inspect and modify any bits that pass through it.

High-Fidelity Logging

Take control of the logs by setting up Python's logging with requests for your regular eavesdropping:

import logging
import requests
from http.client import HTTPConnection

log = logging.getLogger(__name__)  # Get a log grip

logging.basicConfig(level=logging.DEBUG)  # Set the verbosity level

requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

HTTPConnection.debuglevel = 1  # Crank up debugging

Place the code snippet with care in your codebase. A bit like hiding the cookie jar from children. Clever logging configuration leads to easier issue resolution.

Logging Like the Pros

Know your logging levels: DEBUG for development, but maybe INFO or ERROR for production.
To beat noise, filter your logs.
Rotate the logs to avoid filling up your storage.