Download large file in python with requests

Accommodating different content types

Different content types like JSON or binary files require distinct processing methods. Make sure you know your file like the back of your keyboard:

• Binary content recommends the streaming method above.
• JSON data starts a party with r.json(), right after r.raise_for_status().

Confirm the Content-Type header is as predictable as your coffee order by checking r.headers['Content-Type'].

Error-proofing and file names extraction

Prevent unexpected HTTP errors or URL mishaps by introducing r.raise_for_status() immediately after making your request. If you're playing guessing games with the filename, try url.split('/')[-1] - it's fun!

Different strategies for downloading files

The urllib.request alternative

urllib.request.urlretrieve makes for a viable request alternative when emotions aren't involved:

import urllib.request

urllib.request.urlretrieve('http://example.com/largefile.zip', 'largefile.zip')


The temporary file route

Sometimes, we need to stream a response direct to a temporary file. It's easier than it sounds:

import shutil
from requests import get
from tempfile import NamedTemporaryFile

response = get('http://example.com/largefile.zip', stream=True)
with NamedTemporaryFile(delete=False) as f:
    shutil.copyfileobj(response.raw, f)  # Making a RAW deal here!


The wget module option

Always go simple when you can. Using the wget module is like sending a trained retriever (the wget dog) to fetch your frisbee (the large file):

import wget

filename = wget.download('http://example.com/largefile.zip')  # Go fetch, wget!


Optimizing your download

Memory management

Watch memory consumption like an eagle during download and adjust chunk_size for a good balance of speed and efficiency.

Disk write performance

To make sure data is written instantly to disk, f.flush() and os.fsync(f.fileno()) make a compelling duo.

Working with compressed content

When downloading gzipped content, you'll need to manually unpackage it like an eager kid on a Christmas morning:

if r.headers.get('Content-Encoding') == 'gzip':
    file_content = gzip.decompress(r.raw.read())  # Unwrapping a shiny new toy!
