How do I concatenate text files in Python?

Reading large files line by line

When dealing with large files, reading the entire content is about as practical as trying to eat an elephant in one bite. Instead, use a line-by-line approach to take manageable bites:

with open('output.txt', 'w') as outfile:
    for f in ['file1.txt', 'file2.txt']:
        with open(f, 'r') as infile: # Open sesame! Or, you know, the file
            for line in infile: # Line dancing
                outfile.write(line) # Passing the baton


This technique is memory-friendly and efficient for chunkier text files.

The beauty of itertools

If you're dealing with many files and want to look cool doing it, let itertools.chain.from_iterable do the heavy lifting:

import itertools

files = ['file1.txt', 'file2.txt']
with open('output.txt', 'w') as outfile:
    # Behold! The magic of itertools
    outfile.writelines(itertools.chain.from_iterable(map(open, files)))


When dealing with non-text data, open files in binary mode, otherwise, you could corrupt your data. Uncool!

The elegance of fileinput

The fileinput.input() function lets you treat multiple input streams as just one file. It's like seeing the Matrix. Except, instead of dodging bullets, you're iterating through lines:

import fileinput

with open('output.txt', 'w') as outfile:
    for line in fileinput.input(['file1.txt', 'file2.txt']):
        outfile.write(line) # Just another line in the wall


This simplifies file handling while making your code look nifty.

Pattern matching with glob

Using the glob module, you can match patterns like a cop in a crime drama:

import glob

file_pattern = '*.txt' # The Usual Suspects
file_list = glob.glob(file_pattern) # Line 'em up

with open('output.txt', 'w') as outfile:
    for file_name in file_list:
        with open(file_name, 'r') as infile:
            outfile.write(infile.read()) # Bring 'em all in!


This brings order to the chaos of matching filename patterns.

Handing over large files in chunks

With very large files, our best option is to read and write them in chunks. It's like playing with Lego, but you're building a document:

chunk_size = 1024 # Because even Python chews its food
with open('output.txt', 'wb') as outfile:
    for f in ['bigfile1.txt', 'bigfile2.txt']:
        with open(f, 'rb') as infile:
            while chunk := infile.read(chunk_size):
                outfile.write(chunk) # Python, master mason


Efficient concatenation with shutil

The shutil module offers another approach to paranormal file activity. It can help you move the bodies...err, file contents:

import shutil

with open('output.txt', 'wb') as outfile:
    for f in ['file1.txt', 'file2.txt']:
        with open(f, 'rb') as infile: # Breaking the coffins open
            shutil.copyfileobj(infile, outfile) # Haul 'em over
