Understanding generators in Python

python

generator-expressions

lazy-loading

event-driven-programming

byNikita Barsukov·Mar 9, 2025

In Python, a generator is a construction that lets a function produce a sequence of results instead of a single value. When creating iterators, it is often easier and recommended to use generators.

Here's a simple demonstration:

def count_up_to(max):
    for i in range(1, max + 1):
        yield i

for number in count_up_to(5):
    print(number)  # Prints: 1 2 3 4 5, just counting up in style.

A generator is memory-friendly due to its lazy evaluation mechanism, keeping track of state to be able to produce the next value on demand. This makes it ideal for processing large data sequences.

Why should you use generators?

Controlling the flow with `yield`

To meticulously control the flow of your Python program, you can employ generators. When a yield statement is encountered, the state of the generator function is saved and control is relinquished to the caller. On invoking .next() (or next() in Python 2.7), control is returned exactly where it was left, allowing the function to proceed smoothly.

Economizing memory with generators

Dealing with large data has its memory constrictions. A generator helps to step around this problem. It reads, processes and yields one line (or a set of lines) at a time, never bothering to load the entire file into memory.

Unleashing itertools

Go grab your toolkit! Python's itertools module, is your indispensable aid when working with generators. It presents several functions for creating effective looping constructs. Be it chaining iterators or arranging complex permutations, itertools has your back!

import itertools

# An endless merry-go-round
for number in itertools.cycle([1, 2, 3]):
    print(number)  # Loops: 1, 2, 3, 1, 2, 3, ..., or till you get dizzy!

# A stop sign on the path
for i in itertools.takewhile(lambda x: x <= 5, itertools.count()):
    print(i)  # Prints: 0, 1, 2, 3, 4, 5, and then it dutifully stops.

Deep diving into generator magic

Using generator expressions

Generator expressions, akin to list comprehensions, offer a short syntax to swiftly create generators without formulating a full function:

gen_expression = (x**2 for x in range(10))

for value in gen_expression:
    print(value)  # Prints: 0, 1, 4, 9, ..., 81, because squaring is fun!

Perfect for quick iterations, gen-expressions are encased in parentheses (), differing from the square brackets [] for list comprehensions.

Advanced techniques with generators

Generators conveniently encapsulate logic for processing data streams, event-driven programming, and can even infinite loops while savouring coffee breaks.

def event_stream(events):
    while True:
        event = next(events)
        yield handle_event(event)

The event_stream processes events lazily (gotta love lazy Sundays), spitting out results one at a time, without feeling the need to stuff all events up-front.

Interacting with generator objects

Although for loops manage generator objects nicely, going under the hood reveals few nifty options, mainly:

.send(value) - Resume the generator, while also sending a value back into it.
.throw(type[, value[, traceback]]) - Make the generator cough up an exception, while fetching the next sweet value.
.close() - A red signal, raising GeneratorExit into the generator, else bakes a RuntimeError with fresh doughnuts.