A non-blocking read on a subprocess.PIPE in Python

Threading and queuing: Like a boss

Using multithreading for non-blocking I/O can be incredibly effective. Python makes it super easy to spin off a worker thread to manage subprocess output.

Wolfpack Setup: The worker and the queue

Start by defining a worker function and a queue to handle the output:

from queue import Queue, Empty
from threading import Thread
import subprocess, select, sys

# 'POSIX' or 'BUST' - that's the question
ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()


Managing the wolfpack: Starting the worker thread

Next, create your subprocess and kickstart the worker thread:

proc = subprocess.Popen(['your_command'], stdout=subprocess.PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(proc.stdout, q))
t.daemon = True  # The thread is like a phantom - vanishes with the program
t.start()


The daemon = True makes sure the thread won’t linger if the main program finishes before it.

Non-blocking reading: Like eating a hot pizza slice

You can now read from the queue in a non-blocking way, just be careful not to burn your fingers:

try:
    line = q.get_nowait()
except Empty:
    pass  # Go make another round of pizza
else:
    print(line)  # Time to feast on your dearly awaited data


Cleaning up: No leftovers, please!

Don’t forget to close the output stream and wait for the subprocess to finish. Leaving the resources leaking is worse than leftover pizza crusts :

proc.stdout.close()
proc.wait()


It's thread-safe and works under both Windows and Linux. So relax, sit back and have a slice of pizza!

Going Async: Unleashing the beast

The asyncio library can be a lifesaver for IO-bound and high-level network code. You can do some amazing non-blocking reads by defining protocols or using coroutines.

Subprocess management: Orchestrate like a maestro

Start by creating an asynchronous subprocess:

import asyncio

async def run_command(command):
    proc = await asyncio.create_subprocess_exec(
        command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )

    while True:
        data = await proc.stdout.readline()
        if data:
            print("Async beauty:", data.decode('ascii').rstrip())
        else:
            break
    await proc.wait()

loop = asyncio.get_event_loop()
loop.run_until_complete(run_command('your_async_command'))
loop.close()


This coroutine-based method is clean and efficient to handle subprocesses without blocking your main program flow.

Smarty pants: Using os.set_blocking

For the UNIX wizards amongst us, the os.set_blocking function toggles the blocking mode of the specified file descriptor:

import os

# Flip the switch to non-blocking
fd = proc.stdout.fileno()
os.set_blocking(fd, False)


With the non-blocking flag set image, proc.stdout.read() or proc.stdout.readline() won’t block your ambitious code execution and will instead raise an exception if there isn’t data ready. Do catch and handle these exceptions, please.

Extra care: asyncproc for more comfort

Sometimes, you might find the asyncproc module more comfortable:

from asyncproc import Process

myProc = Process(["your_command"])
while True:
    # Check subprocess status, it's more important than checking your Insta feed
    poll = myProc.wait(os.WNOHANG)
    # A gift from subprocess, unwrap it carefully
    (stdout, stderr) = myProc.read()
    if stdout:
        print("Output:", stdout)
    if poll is not None:
        break  # You can now check your Insta feed, the subprocess is done


asyncproc wraps asynchronous process management in a user-friendly object-oriented way.

Always the right dress: Choosing the correct non-blocking strategy

Python version and operating system: These two can indeed be party poopers or party poppers - they can significantly influence what methods you can or should use for non-blocking IO. Stick to built-in libraries and standard modules for compatibility and maintenance ease. And yes, please ensure it looks nice and works well in all different party themes (read: environments).