Showing the stack trace from a running Python application

Breaking into debug mode with signals

On Unix systems, you can perform a quick heist on a running Python application using signals (think of them as audible break-in alarms 🔉).

Just program the listener to crane its ears from the get-go:

import signal
import pdb

def interrupt_handler(sig, frame):
    # Like taking candy from a baby 
    pdb.set_trace()

signal.signal(signal.SIGUSR1, interrupt_handler)


Trigger off the alarm with:

kill -SIGUSR1 <pid>


You'll tumble into an interactive pdb session, allowing you to pry into variables and amble through code execution. Mosey along by pressing control-d.

Be warned though, this method might gatecrash I/O operations and isn't completely undetectable.

Sneaky background processes

For backgrounded Python processes where you need to operate like a silent cat burglar, you need a different MO. An external script might just do the trick.

The guys and gals over at the Python cookbook have a nice recipe for just this scenario (recipe link).

Gdb for a deeply layered cake stack

Didn't place a bug in before the caper started? No problem. gdb (the Grand Debugger) can be your reliable accomplice for such missions:

gdb python <pid>


Just don't forget to pick up the right set of tools - the python-specific gdbinit file. Then, just unleash gdb commands like pystack and examine the stack trace even in the dark.

And in case your system gets pesky about gdb getting cozy with ptrace, you might need to grease its wheels:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope


Pyrasite, the non-disruptive lurker

Pyrasite is the ultimate ghost in the shell. It slithers into a running Python process, injects code, and simply walks out with the stack traces:

sudo pyrasite <pid> dump_stacks.py


It's amazingly stealthy and causes zero disruption. Public Service Announcement: Always ensure it can operate in your specific environment.

Sly stack dumps using threading, sys, and traceback

Sometimes, you need to do the dirty work yourself. Here's the blueprint to implement your own stack dumper:

import threading
import sys
import traceback

def dump_stacks(signal, frame):
    id_to_name = {th.ident: th.name for th in threading.enumerate()}
    code = []
    for thread_id, stack in sys._current_frames().items():
        code.append("\n# Thread: %s(%d)" % (id_to_name.get(thread_id,""), thread_id))
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
    print("\n".join(code))

signal.signal(signal.SIGUSR2, dump_stacks)


This nifty signal handler helps you pull out stacks from an orchestrated jumble of threads.

Handling multi-threaded gumballs

Multi-threaded environments are like herding cats. You must devise a plan regarding thread safety and state consistency while performing stack machinations. Keep a vigilant eye on the state of play before making your move.

The top-secret blueprints: strace

Although not Python-specific, strace is a priceless tool. It provides classified knowledge about system calls and signal deliveries within your Python application. Start peeping into the holes to catch sight of system-level details that your Python program might be hiding.