Actual meaning of 'shell=True' in subprocess

python

subprocess

security-risks

best-practices

byAnton Shumikhin·Jan 27, 2025

Using shell=True in the subprocess module of Python lets you use features such as wildcards, pipes, and environment variable expansion by leveraging the shell. It is powerful for creating complex commands, but beware, always sanitize the inputs as it can lead to a security risk known as shell injection.

Here's an example of using it to count the lines in all '.py' files:

import subprocess
subprocess.run('wc -l *.py', shell=True)  # Count all the lines, one Python file at a time

The shell=True option allows *.py to be expanded by the shell. Without it, you would have to gather the file names in Python manually and pass them to subprocess.run(). So remember, always sanitize your inputs and minimize the use of shell=True to prevent potential security issues.

What does shell=True implies?

Security risk with shell=True

With shell=True, your command string is passed to the shell, opening you up to shell injection if the inputs aren't sanitized. Always remember:

If you can, use shell=False: It's the default setting and a safe option.
Sanitize all inputs: If you do need to use shell=True, make sure all inputs are well-sanitized, especially when they come from untrusted sources.

Performance impact of shell=True

Invoking shell=True can add overhead as it has to invoke an instance of the shell, then run the command within the shell context. This usually isn't an issue, but if you're using shell=True within a loop or calling subprocesses frequently, the overhead might be noticeable.

shell=True on different platforms

Different shell behaviors across platforms can lead to unexpected behavior with shell=True. For Unix-like systems, Bourne shell descendants (like bash) are used, while Windows uses cmd.exe. So, always consider shell availability when using shell=True.

How to harness the power of shell safely

Using subprocess methods directly

You can mimic shell features with native Python functionality, such as:

Redirection: Instead of using > in a shell command, use stdout=subprocess.PIPE within the subprocess method.
Wildcards: Python's glob.glob('*.py') can replicate wildcard functionality.
Piping: Chain subprocess calls with stdout=subprocess.PIPE (the first command), and stdin=first_process.stdout in the subsequent one.

Using shlex and os.path methods

Utilize shlex.split() for parsing command-line strings before sending them to subprocess.run().
Use os.path.expandvars for expanding environment variables, instead of relying on shell interpolation.

Avoiding shell invocation

Always opt for subprocess.run(['command', 'arg1', 'arg2']) without a shell, if you're dealing with simple commands. For complex requirements, consider the security risks and implement strategies to maintain security.

Advanced subprocess features

Dealing with output and errors

subprocess.check_output() captures output from a command and raises an error if it fails, unlike subprocess.run().
While using shell=True, ensure paths with spaces or special characters are well escaped to avoid unintended shell expansions.

Controlling the subprocess environment

Modify the env parameter in subprocess.run() calls to manage environment variables of the subprocess.
When combining commands, avoid shell=True and use Python's native itertools or functools for in-memory transformations.