Extracting specific selected columns to new DataFrame as a copy

Go deeper: Extracting columns without headaches

You know the basics now, but let's step up our game. Here comes some advanced maneuvers for column extraction.

Use .copy() to avoid complications

When creating a new DataFrame that's a subset of an existing one, append .copy() to escape the dreaded SettingWithCopyWarning. This makes a deep copy of your data, preventing any unintended consequences in your original DataFrame.

# Copy columns A,B & C. Now df2 is a standalone dataframe(copy of df1) and won't bite you back!
df2 = df1[['A', 'B', 'C']].copy()


Fancy filtering

What if you want to play favorites with column names? Use the filter() function.

# Here, df.select_dtypes(include=[np.number]) selects all numeric columns. Don't include any column you're not on first name basis with!
new_df = df.filter(df.select_dtypes(include=[np.number]).columns)


Dropping columns like they're hot

Sometimes dropping a column can feel as if you're on a mission to disarm a time bomb. Be calm, the drop() method is here to rescue!

# The column B's been acting up? Just drop it. We don't need that negativity (B column) in our life.
new_df = df.drop('B', axis=1)


Spy mode: Selecting by index with iloc

In circumstances where columns are under incognito (names aren't reliable or known), use the iloc method to select columns by position.

# You don't know me by my name. But, you know where to find me.
new_df = df.iloc[:, [0, 2, 3]].copy()  # Assuming 'A', 'C', 'D' are the first, third and fourth columns.


Making pandas handle memory more efficiently

When you're dealing with colossal DataFrames, it's not time and resources friendly to drop unwanted columns. filter() or iloc() methods are more gutsy for large DataFrames.

The Real-world playbook for column selection

Time for real game time decisions. Let's get more practical with your column selection techniques.

Efficiently selecting columns by data types

A real character of a savvy programmer is efficiency. Easily select columns of a certain type, such as integers, by making use of select_dtypes():

numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()
new_df = df[numeric_cols]


Evading the memory hogs

If you're working with a DataFrame bigger than King Kong, you want to make sure your computer memory is not getting wiped out. Hence, selecting a few columns to keep instead of dropping a large number may result in a bit more memory-friendly code:

cols_to_keep = ['A', 'B', 'F']  # B, F are for Bigfoot
new_df = df[cols_to_keep]


Snappy dynamic column selection

Let your columns dance on your fingertips. Select columns dynamically based on conditions:

# Your friend: Hey, can you grab me all columns that contain 'sales'?
# You, a Python lord: Sure, no biggie!
cols_to_select = [col for col in df.columns if 'sales' in col]
new_df = df[cols_to_select].copy()
