Search and replace a line in a file in Python

Managing chunky data gracefully

In scenarios where the files are extremely large, maintaining low memory usage is essential. One solution is to process the file line by line. By using the tempfile module combined with shutil for file operations, you can replace data without risking truncation or memory flooding:

import shutil
from tempfile import mkstemp
import os

# In space, no one can hear your....wait, wrong line
fh, abs_path = mkstemp()
try:
    with os.fdopen(fh, 'w') as new_file:
        with open('file.txt', 'r') as old_file:
            for line in old_file:
                new_file.write(line.replace('old_line', 'new_line'))
    # Perfect copy of permissions from the old file.
    shutil.copymode('file.txt', abs_path)
    # Old and Busted? Begone!
    os.remove('file.txt')
    # A new file waltzes in
    shutil.move(abs_path, 'file.txt')
except Exception as e:
    # Comic book villain laugh: Muahaha! Caught you, error!
    print(f"An error occurred: {e}")


This script manages the entire process of file modification while owing permissions from the original file and removing the old file after replacement.

Line by line editing

The fileinput module with inplace=1 is a remarkable tool for effective line-by-line in-place editing. However, keep in mind that fileinput ingeniously redirects STDOUT to the input file when inplace=1, requiring the use of the print() function for writing changes:

import fileinput

filename = 'file.txt'
backup_extension = '.bak'

for line in fileinput.input(filename, inplace=1, backup=backup_extension):
    # Abracadabra! Line changed!
    print(line.replace('old_line', 'new_line'), end='')


The backup file ensures the availability of an original version of the content, providing a safety layer in case a replacement operation goes adrift.

Applying production awareness

In real-world coding scenarios, explicitness is a virtue. Defining explicit functions for search and replace operations improves code clarity. Plus, always ensure to use a context manager to guarantee that files are correctly closed:

def search_and_replace(file_path, search_text, replace_text):
    with fileinput.FileInput(file_path, inplace=True, backup='.bak') as file:
        for line in file:
            # Voila! You witnessed a transformation
            print(line.replace(search_text, replace_text), end='')

try:
    # Just another day of search and replace!
    search_and_replace('file.txt', 'old_line', 'new_line')
except Exception as e:
    # Caught an error, some points for me!
    print(f"Did you break something?? {e}")


Encasing the logic in functions and using context managers makes your code compliant with best coding drill and primes it for practical applications.

Using the power of regex

For more complex string replacements where you're dealing with patterns, Python's re module is your best buddy. Use re.sub for multiline regex replacements based on a pattern:

import re

pattern = r'old_line'
replacement = 'new_line'

with open('file.txt', 'r') as file:
    # We walk a lonely road, the only one that we have ever known...
    content = file.read()
# Presto! The lines switch!
content_new = re.sub(pattern, replacement, content)

with open('file.txt', 'w') as file:
    # Don't worry, it's not goodbye, it's "See you soon!"
    file.write(content_new)


This setup converts the pattern into a regex object and replaces all instances of the pattern in just a snap. It's mind-blowingly efficient, especially when dealing with complicated search and replace tasks.

Encountering potential surprises

Anticipate potential problems and evade unexpected encounters. Some probable surprises:

• Race conditions: Code defensively for file operations in concurrent environments. Ensure only one part of the code accesses a file at a time to avoid race conditions.
• Character Encoding: Watch out for the file's encoding, or you might end up with scrambled text. Specify encoding='utf-8' when opening a file.
• Newlines: Python's universal newline support may alter newlines when reading and writing. Prevent this by using newline='' when calling open().
• File permissions: Cautiously check for access privileges when working with sensitive files. This evades permission errors and guards against security vulnerabilities.