How to join two generators (or other iterables) in Python?

python

generator-expressions

lazy-evaluation

iterators

byNikita Barsukov·Feb 19, 2025

itertools.chain() is the function you need when you wish to combine two or more iterables in Python into a single, continuous results set. It works seamlessly for both generators and other iterables.

Let's dive into an example:

from itertools import chain

# Imagine gen_numbers as a pizzas and gen_letters as beers
gen_numbers = (pizza for pizza in range(3))  # 🍕
gen_letters = (beer for beer in 'ab')  # 🍺

# Combine those delicious items with chain
combined = chain(gen_numbers, gen_letters)

# Time to consume! Let's eat and drink
for item in combined:
    print(item)  # Result: 🍕, 🍺

Notice that chain() doesn't munch on all the pizzas and beers in one gulp. Item consumption is lazy, just like a chilled Sunday.

Delegating to sub-generators with yield from

Python 3.3 brought with it a neat feature called yield from, providing us a sweet way to delegate operations from one generator to another. It comes into its real strength with Python 3.5+, where we can use it deftly to join multiple generators.

def combined_generators(gen1, gen2):
    yield from gen1  # I'll take care of the first part
    yield from gen2  # Your turn, buddy!

# Usage
for item in combined_generators(gen_numbers, gen_letters):
    print(item)  # Same result as the lazy Sunday

Here we cleverly avoided chaining our iterables directly and went for a delegated approach.

Speaking concisely with Generator Expressions

When stemming from a desire for an inline solution or a case where chain() just doesn't cut it, generator expressions rise to the challenge very nobly. They let us nest loops right within their syntax, all in a single line.

Time for a quick example:

combined = (item for iterable in (gen_numbers, gen_letters) for item in iterable)

# Print time! Spoiler alert, the result is same
for item in combined:
    print(item)

This could be your best friend when dealing with complex cases needing nested loops or custom checks.

Pitfalls to sidestep

Just a word of caution here, Python says big NO to concatenating generators with +. Also, refrain from converting generators into lists too early. After all, generators are meant for producing items lazily, remember? Always keep in mind, we don't want to gobble the pizza before it's served!

Dive deeper: Handling complex data structures

For situations where you need to merge more complex data structures, like directories or files, Python equips you with itertools.chain.from_iterable(). Specifically effective if you're dealing with situations like folder structure traversals.

Check this out:

from itertools import chain

# Let's pretend we're combining files from different folders
files_folder1 = ['file1.txt', 'file2.doc']  # Folder 1 
files_folder2 = ['file3.pdf', 'file4.ppt']  # Folder 2

# Club them together into one pile
all_files = chain.from_iterable([files_folder1, files_folder2])

# Now, let's read through 
for file in all_files:
    print(file)  # Will print: 'file1.txt', 'file2.doc', 'file3.pdf', 'file4.ppt'

Dealing with iterative complexities

When the task at hand is beyond basic chaining, like applying custom filtering or transformations, custom generator functions come to our rescue. They allow us to tailor the behavior we want to impart while iterating over different data sets.

Take a look at this:

def filtered_data(gen, filter_value):
    for data in gen:
        if data > filter_value:
            yield data  # Only yielding if data is bigger than the filter value

# Usage scenario
for item in filtered_data(gen_numbers, 1):
    print(item)  # Will print 2 and 3 (If generators are not exhausted before)