Http requests and JSON parsing in Python

python

http-requests

json-parsing

requests-library

byAnton Shumikhin·Jan 17, 2025

Quickly fire off HTTP requests and perform JSON parsing using the requests module in Python:

import requests

# Send HTTP GET request (like a ninja) and parse JSON into a dictionary (like a magician)
response = requests.get('https://api.example.com/end-point')
data = response.json()

Replace 'https://api.example.com/end-point' with your desired API endpoint. Make sure the request is successful before digging into your data treasure chest.

Comprehensive guide

Making HTTP requests and processing the JSON response are bread and butter of many Python applications. With the unprecedented power and elegance of Python's requests library, these tasks become a breeze.

Crafting artisanal HTTP requests

Need more spice in your requests? Feed your params to requests.get(), because it loves munching on dictionaries. For constructing the URL, say bye-bye to messy string concatenations:

params = {'origin': 'Narnia', 'destination': 'Hogwarts', 'waypoints': 'Middle Earth|Mordor', 'sensor': 'false'}
response = requests.get('https://maps.googleapis.com/maps/api/directions/json', params=params)

Protip: waypoints value is one string. It's like a road trip: 'Middle Earth|Mordor'.

Gently unwrapping the JSON gift

After your request, get the JSON response using the built-in JSON decoder. Don't forget to check if everything's okay because sometimes life doesn't go as planned:

data = response.json()
if response.status_code == 200:
    from pprint import pprint
    pprint(data) # Who doesn't like nicely arranged stuff?
else:
    print("HTTP request failed harder than my diet:", response.status_code)

Taming the wild JSON data

A JSON response can be a nested chaos. But fear not, using Python you shall conquer:

for route in data['routes']:
    for leg in route['legs']:
        for step in leg['steps']:
            print(step['html_instructions'])

Dive into the JSON. It's just loops... all the way down.

Important points to etch on your mind (or a sticky note)

Develop a habit of validating the response status — API calls can fail...like my attempts at humor.
Stick to .json() but when feeling adventurous, try json.loads(response.content).
Don't be stingy on parameters: origin, destination, and sensor — the more the merrier.

Master strokes with the requests library

Covering basics was fun, but the real magic of requests library lies in its advanced features:

Sessions: Keep requests.Session() in mind, it's commonly used but often misunderstood.
Headers: It's time to play dress-up. Simulate different browsers, handle authentication.
Timeouts: Because waiting indefinitely is the worst.

Troubleshooting guide: What to do when the sky falls down

Error Codes: Knowing your HTTP status codes can save a lot of guesswork.
Exceptions: Protect yourself with try-except blocks, like your personal error-proof armor.
Debugging: Learn to love logging. It might just save your life (or at least, a lot of time).

Juice up the performance

For those who are not just satisfied with "it works" — how can we optimize?

Connection Pooling: Session objects in requests are the secret weapon here.
Asynchronous Requests: Heard about aiohttp? Time to take that speed to the next level.
Stream Large Responses: Save memory by turning on stream parameter, one chunk at a time.