Get a filtered list of files in a directory

python

file-system

pattern-matching

regular-expressions

byAlex Kataev·Dec 9, 2024

To expediently fetch a list of files in a directory following a specific pattern, os.listdir() coupled with list comprehension comes in handy:

import os

# Directory and pattern to look for
directory = '/path/to/the/directory'
pattern = '.txt'

# Get all .txt files
filtered_files = [f for f in os.listdir(directory) if f.endswith(pattern)]

print(filtered_files)

In the above block, we are cherry-picking .txt files from directory. By changing pattern, you can filter files of different types.

Filtering files using glob

glob, Python's own module, dramatically simplifies the process of pattern matching, providing an interface almost akin to our beloved command-line:

import glob

# Bounty Hunt: All .jpg files prefixed with '145592'
filtered_files = glob.glob('/path/to/directory/145592*.jpg')  # Like finding Waldo!

print(filtered_files)

By employing glob.glob() instead of os.listdir(), we're able to bypass the time-consuming for-loop, making our search for Waldo (or jpg files) quicker!

Flexible pattern matching with regular expressions

For the moments when wildcard just doesn't cut it - Regular expressions to the rescue, donning the cape of re module!

import os
import re

# Gear up: Files that start with numbers and end with .jpg
pattern = re.compile('[0-9]+.*\.jpg')   # Pattern: Because randomly searching isn't cool.

directory = '/path/to/directory'
filtered_files = [f for f in os.listdir(directory) if pattern.match(f)]

print(filtered_files)

Complex patterns require a complex hero. Enter regular expressions!

Regex meets glob: Dream team?

For patterns whose complexity knows no bounds, club glob with re to form the ultimate crime-fighting team:

import glob
import re

# Ultimate Quest: Files matching a Regex pattern
pattern = re.compile('[0-9]+.*\.jpg')
all_files = glob.glob('/path/to/directory/*')

# Pattern fighting crime one file at a time!
filtered_files = [f for f in all_files if pattern.match(os.path.basename(f))]

print(filtered_files)

Balancing performance and code cleanliness is key, especially when we have large file sets lurking in the shadows!

Filtering for specific file types

Using fnmatch.filter() we can get specific files based on their type:

import os
import fnmatch

directory = '/path/to/your/directory'
filtered_files = fnmatch.filter(os.listdir(directory), '*.py')  # Hunting down those pythons!

print(filtered_files)

Just like us, Python too gets overwhelmed when too much is happening at once. Let's give Python a breather using os.scandir() and generators:

import os

# Define a generator to breathe easy while sifting through .log files
def get_log_files(directory):
    with os.scandir(directory) as entries:
        for entry in entries:
            if entry.is_file() and entry.name.endswith('.log'):
                yield entry.name  # One .log file at a time, there's no rush!

directory = '/path/to/large/directory'
filtered_files = list(get_log_files(directory))
print(filtered_files)

Make sure the folder path in your code is accurate, else you'll possibly end up courageously battling errors instead of filtering files!
Handle string to list conversion like raw eggs — carefully to avoid unexpected and chaotic explosions.
When dealing with large file lists, memory usage can often be overlooked — don't let it sneak up on you.
Python gives us a lot of good stuff out of the box — plugins, modules — make efficient use of them for code efficiency and easier maintenance.