Using headers with the Python requests library's get method

python

http-headers

requests-library

python-requests

byAnton Shumikhin·Mar 9, 2025

To perform a GET request with custom headers via the requests.get method, inject a headers dictionary:

import requests

url = 'https://example.com'
headers = {'User-Agent': 'Your Bot', 'Accept': 'application/json'}

response = requests.get(url, headers=headers)

Alter 'https://example.com' to your desired URL, and equip the headers dictionary with your custom headers. Make your User-Agent unique and specify to the server that you're expecting a JSON response.

HTTP headers: Usage and management in requests

What are HTTP headers and why do they matter?

HTTP headers serve as a transmission control for your web-based requests. They're basically the Post-it notes for the HTTP transaction, providing additional parameters and guidance (for instance, delivery settings for an international parcel). Popular headers include User-Agent, Accept, Content-Type, and more.

Making your requests count with custom headers

Custom headers are often utilized for authentication keys, content specification, or to fulfill CORS requirements. With the requests library, you can easily append and adjust headers, ensuring your requests get through the screener:

headers = {
    'Authorization': 'Bearer JWT_TOKEN_IS_HERE',  # Add yours here, no peeking!
    'Accept-Language': 'en-US',
    'Content-Type': 'application/json'
}

response = requests.get('https://api.example.com/endpoint', headers=headers)

Sustained headers with sessions

For keeping headers across multiple requests, consider using a Session object. This makes you less repetitive, like a great DJ with consistent transitions:

session = requests.Session()
session.headers.update({'User-Agent': 'Pythonista Bot'})
response = session.get('https://api.example.com/data')

Sessions also manage cookie persistence, making it a useful ally for server interactions requiring state maintenance.

Dealing with common header fields

Apart from Authorization, these are some common fields:

User-Agent: A way to identify your request's client, be it a browser or a personal crawler.
Accept: Clear communication with the server about what format you'd like your data in.
Cookie: Especially handy if you want your request to be personalized or use existing session data.

Expert manipulation of headers

This is where you take control for complex requests. The pro tip here is to refer to the official documentation regularly to understand the nuances. A wrong Accept field might fetch you data in Klingon (🖖) instead of plain English!

Best practices and tips for pro users

Error management during header usage

Error handling is an essential part of any request-related code. Implementing it ensures that header-related errors get notified instead of quietly crashing your app:

try:
    response = requests.get('https://api.example.com/resource', headers=headers)
    response.raise_for_status()
except requests.exceptions.HTTPError as err:
    print(f'HTTP error occurred: {err}')

Timeouts and authentication for added security

A timeout parameter can act as a watchdog, preventing your request from hanging indefinitely:

response = requests.get('https://example.com', headers=headers, timeout=5)

For HTTP Basic Auth scenarios, just use the auth parameter:

response = requests.get('https://example.com', auth=('username', 'password'))

Harnessing parameters and cookies for advanced data handling

The params argument in the get function can help send data easily, just like passing cookies through the cookies argument:

params = {'key1': 'value1', 'key2': 'value2'}
cookies = {'session_id': '123456789'}
response = requests.get('https://example.com/search', headers=headers, params=params, cookies=cookies)