Wait until all threads are finished in Python

python

threading

concurrent-execution

exception-handling

byNikita Barsukov·Mar 10, 2025

To effectively manage and wait for multiple threads in Python, create threading.Thread for each task, and deploy .join() for all threads, ensuring their complete execution. Conveniently handle them via a list.

Example:

import threading

def task():
    print("Thread working")  # Hard worker, indeed

threads = [threading.Thread(target=task) for _ in range(5)]  # Squad of threads ready!
for t in threads: t.start()  # Release the Krakens!
for t in threads: t.join()  # Wait for the troops to return

print("Threads done.")  # Mission accomplished!

The .start() action fires up the thread while .join() effectively makes the main thread pause and wait until every worker thread has finished.

Leveraging Python's ThreadPoolExecutor

Managing threads individually can be somewhat mind-boggling, more so with large numbers. The built-in Python library concurrent.futures.ThreadPoolExecutor brilliantly handles this, offering a superior method of asynchronous task execution, thus easy thread management.

A mini demonstration:

from concurrent.futures import ThreadPoolExecutor

def task():
    print("Thread working")  # Busy bee here!

executor = ThreadPoolExecutor(max_workers=5)
futures = [executor.submit(task) for _ in range(5)]  # You're charged, my knight!
for future in futures:
    future.result()  # Sweet fruits of labor!

print("All threads have finished.")  # Endgame

You can limit the number of threads using the max_workers option, ensuring your system doesn't cough under the weight of excessive threads.

Object-Oriented Threading

In more complex applications, it pays off to define classes for managing threads and encapsulating behaviors. It enhances structure and scalability, enabling robust, maintainable, and efficiently testable systems.

Illustration of Custom thread class:

import threading

class WorkerThread(threading.Thread):
    def __init__(self, worker_id):
        super().__init__()  # Like father, like son
        self.worker_id = worker_id

    def run(self):
        print(f"Thread-{self.worker_id} working")  # At your service, Lord Commander!

threads = [WorkerThread(i) for i in range(5)]  # My own army!
for t in threads: t.start()  # Charge!
for t in threads: t.join()  # Come back soldiers!

print("All custom threads have finished.")  # The battle is won!

By taking advantage of threading.Thread subclassing, we birth specialized threads endowed with additional data or behaviors.

Dipping into multiprocessing

For CPU-bound tasks seeking true parallelism, Python's multiprocessing module is a knight in shining armor. Every Process instance jets off in a separate Python interpreter, guaranteeing concurrent task execution.

Basics, to the point:

from multiprocessing import Process

def task():
    print("Process working")  # Look ma, no hands!

processes = [Process(target=task) for _ in range(4)]  # Process quartet is ready!
for p in processes: p.start()  # Get set, go!
for p in processes: p.join()  # Hold on, let’s regroup!

print("All processes have finished.")  # End of an era

With multiprocessing, you can offload work to different CPUs, perfect for computation-heavy tasks.

Shape up or ship out: Handling exceptions in threads

Exception handling is fundamental for robust threading. Make sure your thread functions are equipped with try-except blocks to capture and handle exceptions. This prevents a single thread's hiccup from spiraling into a catastrophic system crash.

Demo with exception handling:

import threading

def task():
    try:
        # Thread work starts here
        print("Thread working")  # Sweating bullets here!
    except Exception as e:
        print(f"A wild error appears: {e}")  # Panic at the disco?

threads = [threading.Thread(target=task) for _ in range(5)]  # The army has risen!

for t in threads: t.start()  # To infinity and beyond!
for t in threads: t.join()  # Let’s regroup, soldiers!

print("Finished with robust exception handling.")  # Dodged bullets like Neo!