Python pandas Filtering out nan from a data selection of a column of strings

Spotlights on NaN expulsion techniques

Single vs Multiple Columns: Choose your battlefield

When your battlefront extends to several columns, aim to remove any row that plays host to NaN in all or any of the specified columns:

# NaN-Repellent: Vanquishes rows nurturing NaN in either 'column1' or 'column2'
cleaned_df = df.dropna(subset=['column1', 'column2'])

# Paranormal Activity Detector: Boots out rows where both 'column1' and 'column2' are haunted by NaN
fully_cleaned_df = df[df[['column1', 'column2']].notna().all(axis=1)]


Exploiting Query Method for Swift Cleanup

Use query to pick a sweet apple from the tree, eschewing those infested by the nan-worm:

# This code makes a sweeping declaration - "No NaNs or 'N/A' allowed!"
df_cleaned = df.query("column_name.notna() & column_name != 'N/A'", engine='python')


Root out disguised 'NaN' trespassers with RegEx

Beware! Some NaNs come incognito as 'N/A' or empty, ghost-like strings:

# 'Fear no more!' says RegEx, 'I shall root out the NANguise!'
nan_filter = df['column_name'].str.match(r'^(?!$|N/A).*$', na=False)
df_filtered = df[nan_filter]


Custom Filtering: Bringing out the big guns with list comprehensions

Big problems need big solutions. List comprehensions are heavy-duty machinery when you need to tackle multiple conditions or run a custom function:

# A forcefield that keeps out non-strings or ill-fitting suits (blanks)
filtered_df = df[[not pd.isnull(x) and x.strip() != '' for x in df['column_name']]]


Advanced Data Ninja techniques

Custom Placeholders: NaN in disguise

Not every NaN is as obvious. They may be masquerading as an innocent '--' or a nondescript 'unknown':

# Unmasking the NANguise
placeholders = ['', 'N/A', 'unknown', '--']
filtered_df = df[df['column_name'].apply(lambda x: x not in placeholders)]


Nullable Integer types: NaN's favorite hideout

Review nullability of your integers. While every integer is proud of its value, some are shy and hide behind a pd.NA mask:

# Stop 'int' from going undercover as pd.NA
df = df.convert_dtypes()


Nullable String types: NaN's secret lair

The newer StringDtype in pandas is a secure vault to lock up the pd.NA:

# Keeping a tab on pd.NA's whereabouts
df['column_name'] = df['column_name'].astype('string')
