How to use subprocess command with pipes

Shell game: When to use shell=True

Despite the tempting tango of danger and darker arts associated with shell=True, it does simplify those commands that are already accustomed to the smooth dance of shell features, like pipes, juggling wildcards and such smooth moves. Know your tango steps, sanitize your inputs, and shell=True will lead the dance:

import subprocess

# Dancing command with a pipe
output = subprocess.check_output("ls | grep py", shell=True)

print(output.decode('utf-8'))


Remember, no 'shell=True' if you're receiving a mysterious client—the untrusted input.

Efficiency pops with subprocess.run

The subprocess.run() command is like the VIP lounge of APIs on Python 3.5, letting you slurp up output directly:

import subprocess

result = subprocess.run(['grep', 'py'], capture_output=True, text=True)
print(result.stdout)


To rage against the dying of the light, or rather, to raise exceptions when subprocess performs its swan song, use check=True:

try:
    subprocess.run(['false'], check=True)
except subprocess.CalledProcessError:
    print("Oops, Houston, we got a problem.")


Output handling like a pro

Encode enigma, aka decode the byte stream to transform it into recognizable str:

output = subprocess.run(['ls'], stdout=subprocess.PIPE).stdout
print(output.decode('utf-8'))


To prevent being caught in endless tango, you can use communicate() to rescue yourself. It's your safety net - preventing deadlocks when capturing stdout and stderr:

from subprocess import PIPE, Popen

p = Popen(['your_cmd'], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()


Hopping between multiple processes

Command training—each learning from the previous—is about closing stdout of the predecessor:

from subprocess import Popen, PIPE

ps = Popen(['ps', '-A'], stdout=PIPE)
grep = Popen(['grep', 'process_name'], stdin=ps.stdout, stdout=PIPE)
# Let 'ps' retire, it's too old for this
ps.stdout.close()

output = grep.communicate()[0]
print(output.decode('utf-8'))


Simplify your life with sh.py

The sh documentarian creates an easier shooting script for subprocess scripting. Get it on set with pip install sh. Directing a pipe becomes as easy as action, cut:

from sh import grep, ps

# Emulate 'ps -ax | grep process_name'
print(grep(ps("-ax"), "process_name"))


Subprocess tricks of the trade

Piping through subprocess is cooking, good cooking. Don't get burned. Here's your chef's tips and tricks:

• Beware of the buffer overflow; large outputs can really stop your kitchen.
• Environment variables are like secret spices that behave in unexpected ways with shell=False.
• Keep the kitchen tidy. Cleaning up zombie processes demands a clean-up of all pots and pans (pipes).

Security is no joke

When playing builder with subprocesses, especially wearing the shell=True hat, hard hats (security best practices) are essential:

• Sanitize those bricks (user input) to prevent a shell injection disaster (attacks).
• Use a reliable vendor (lists) with Popen where you can.
• Use an accredited supplier with shlex.split for safe input intended for shell=True operations.

How to be an efficient builder

Ensure your house (subprocess implementation) is solid, secure, and efficient:

• Use run with capture_output=True or stdout=PIPE for those small projects.
• For big commercial jobs (larger outputs), write to a file or process in manageable chunks.
• Directly creating Popen objects is like hiring a capable manager for complex pipeline control.

More tips and tricks for the expert builder

Savor these additional insights to achieve bricklaying mastery:

• Context managers can lead your Popen workforce effectively.
• Use stderr=subprocess.STDOUT to let your site supervisor (stderr) double as a builder (stdout).
• Study those blueprints (utilize return codes) to check if construction went as planned.