Basic http file downloading and saving to disk in python?

python

file-downloading

streaming

performance-profiling

byNikita Barsukov·Jan 30, 2025

Securely download and save a file from a web URL using Python's requests library with a few clear lines of code:

import requests

response = requests.get('http://example.com/file.txt')
response.raise_for_status()  # If your code reaches here, congrats! The download succeeded.🎉

with open('local_file.txt', 'wb') as file:  # Let's name it local_file and store it.
    file.write(response.content)

This set-up fetches 'http://example.com/file.txt' to your local system, saving as 'local_file.txt'. If download fails, exception handling kicks in.

Tackling large files download

For handling massive downloads, it's preferable to stream to manage memory usage:

import requests

response = requests.get('http://example.com/largefile.zip', stream=True)
response.raise_for_status()  # Fingers crossed till this line. 🤞 

with open('large_file.zip', 'wb') as file:
    for chunk in response.iter_content(chunk_size=8192):
        file.write(chunk)

Voila! Large file successfully downloaded without crashing your system memory.

Dealing with archive files

In scenarios where the download is a .gz type file, Python's gzip module helps unpack after the saving process:

import gzip
import shutil

with gzip.open('file.gz', 'rb') as f_in:
    with open('file.txt', 'wb') as f_out:
        shutil.copyfileobj(f_in, f_out)

Are you thinking what I'm thinking? Yes, you just decompressed a gzipped file. Bursting balloons is no longer the only way to decompress.

Using `requests` like a pro

requests library serves your code dish with some side features like error handling and session objects:

from requests.exceptions import HTTPError

try:
    with requests.Session() as session:
        response = session.get('http://example.com/file.txt')
        response.raise_for_status()  # A successful download response is the new "Bazinga!"

        # Further mystical code...

except HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')  # Printing the error message. No magic here!
except Exception as err:
    print(f'An error occurred: {err}')  # Broad except clause. Don't tell a Python purist.

Alternatives for the adventurous

wget offers a really simplistic approach to file downloads:

import wget

file_url = 'http://example.com/file.txt'
file_name = wget.download(file_url)

One command to rule them all! wget got your back here.

PIL magic with images

Working with images? PIL library can be your magic wand to open and save images:

from PIL import Image
import requests
from io import BytesIO

response = requests.get('http://example.com/image.png')
image = Image.open(BytesIO(response.content))
image.save('local_image.png')

Just like that you went from 'Image? What image?' to 'Image? No problem!'

Python version compatibility check

Ensure your code's compatibility with your Python version. urllib.request.urlretrieve is the Python 3 approved way:

import urllib.request

local_filename, headers = urllib.request.urlretrieve('http://example.com/file.txt')

Finally, a method to match the modernity of Python 3.x.

Benchmarking performance through profiling

Got a giant file to download or doing it repeatedly? Profiling your code can help identify performance inefficiencies:

import cProfile

def download_file(url):
    # Insert your well-crafted download code here...
    ...

cProfile.run('download_file("http://example.com/largefile.zip")')