Running shell command and capturing the output

python

subprocess

shell-execution

error-management

byNikita Barsukov·Sep 13, 2024

Python has a nifty module, subprocess, that works like magic. With the subprocess.run() function, you can execute shell commands and capture their output:

import subprocess

# 'echo' says hi, Python responds 👋
result = subprocess.run(['echo', 'Hello, Python!'], capture_output=True, text=True)
# Printing the magic words
print(result.stdout.strip())

This ecosystem of the subprocess.run() function and the capture_output, sends our messages from echo through stdout, right into the arms of Python.

Shell execution essentials

Converting byte strings

Byte string got you scrunching your eyebrows? Python's .decode('utf-8') function is here to the rescue. It converts the byte string into a human-readable string:

result = subprocess.run(['echo', 'Hello, Python!'], stdout=subprocess.PIPE)
print(result.stdout.decode('utf-8').strip())

Error management

Want Python to catch errors that occur during shell command execution? The check=True parameter has your back:

try:
    result = subprocess.run(['false'], check=True)
except subprocess.CalledProcessError as e:
    print("Woops, a wild error appears: ", e)

Command argument splitting

Got a shell command string and need to make a list out of it? shlex.split() is your genie:

import shlex

command = 'ls -l'
subprocess.run(shlex.split(command), capture_output=True)

Real-time results with Popen

Want to dive deep into the ocean of shell command execution? Let's embark on the exciting journey of subprocess.Popen():

with subprocess.Popen(['ping', 'stackoverflow.com'], stdout=subprocess.PIPE, text=True) as proc:
    for line in proc.stdout:
        print("Pinging: ", line.strip())

Advanced subprocess interactions

Managing errors and outputs together

To lure the outputs and error messages in the same trapping box, use stdout=subprocess.PIPE, stderr=subprocess.PIPE:

com = subprocess.run(['ls', '-l', '/nonexistent_directory'],
                      stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("Gotta catch 'em all, right? Here's the stderr output :", com.stderr.decode())

Process data exchange with communicate()

Reusing the magical lamp of subprocess.Popen() lets you send data to stdin, and then read from stdout or stderr as if you were casually chatting with a process:

process = subprocess.Popen(['cat'], stdout=subprocess.PIPE, text=True, stdin=subprocess.PIPE)
stdout, stderr = process.communicate('Hello, Popen!\n')
print("Exchanging pleasantries with Popen: ", stdout)

Clean resource allocation with `with`

And the with statement, the unsung superhero, ensures all file and process resources are handled cleanly and efficiently:

with subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE) as proc:
    print("Show me the directory details, pretty please: ", proc.stdout.read())

Quick and straightforward tasks

Direct output with `getoutput()`

Presenting the subprocess.getoutput() function, the honey-badger of the Python world. It gets things done without any fuss:

output = subprocess.getoutput('ls -l')
print("Honey-badger results: ", output)

The veteran `check_output`

To ensure backward compatibility, let's bring in the experienced fighter, subprocess.check_output():

output = subprocess.check_output(['ls', '-l'], text=True)
print("Old is gold: ", output)

Error management and safety tips

Exception shielding

To protect your code castle from unknown enemy attacks (errors), raise the drawbridge by implementing try / except blocks:

try:
    subprocess.run(['rm', '-rf', '/important_data'], check=True)
except subprocess.CalledProcessError as error:
    print("Enemy attack deflected: ", error)