Get list from pandas dataframe column or row?

Into the matrix: Converting Series to lists

DataFrames host a congregation of Pandas Series, and converting these Series to lists opens doors for powerful list operations. You need to be aware of missing values (NaN) and data type inconsistencies which can play spoilsport in your code.

Dealing with the vanishing values

Missing values are like Voldemort in your DataFrame, they exist but you can't quite pin them down:

# Evicting Voldemort from your DataFrame
col_without_nan_list = df['B'].dropna().tolist()  # Exorcising NaN values


Conserving magical diversity

Remember how the wand chooses the wizard in the Potterverse? Similarly, the data types choose their operations in coding:

# Grasp the magical essence of your column
print(df['C'].dtypes)

# Transfigure while conserving the essence
col_array = df['C'].to_numpy(dtype='int32')  # Elucidate dtype as per your need


Elucidating advanced extraction techniques

Embracing the conversion of DataFrames into lists or arrays catalyzes unmatched data manipulation powers. Leap deeper into the rabbit hole to learn more about these techniques, for learning is the eye of the mind.

Speed: The master of time

If the sands of time pass slower than your DataFrame loads, .tolist() is your ally. It’s faster and knows how to handle different data types.

import perfplot

perfplot.show(
    setup=lambda n: pd.Series(range(n)),
    kernels=[
        lambda s: s.tolist(),  # The Flash of Python
        lambda s: list(s)  # The Sloth of Python
    ],
    n_range=[2**k for k in range(20)],
    xlabel='Series size'
)


Eventualities unravelled

If your detour involves meeting only unique values, the unique() function will get you exactly there:

unique_values = df['D'].unique()  # The road less travelled


And yes, you can use np.unique(), but just like your ex, it behaves differently.

The row less traversed

Seldom do we need to deal with whole rows, but you know what they say - “expecto the unexpected”:

# Convert Row 3 into a list
row_3_list = df.iloc[2].tolist()  # Why is 6 afraid of 7? Because 7 8(ate) 9.


This technique is the prophecy that foretells saving the day when the hour of converting complete rows into lists comes.

Matrix reloaded

It’s not all doom and gloom when stuck in list or array form. The DataFrame constructor is your key to escape the matrix:

# Transform matrix into DataFrame
df = pd.DataFrame(data=matrix)  # Stepping outside the matrix


Optimization, the final frontier

Before you decide to venture off into the wilderness, think about the data type of your arrays and lists. Depending on what you’re working with, one method might work better than the other.

Scalar type subtleties

The devil lies in the details, or in this case, in the difference between int32 and int64 data types:

# Esoteric subtleties in data conversion
colo_list = df['E'].astype('int32').tolist()  # All magic comes with a price


For more insights, take a trip to Diagon Alley (GitHub) for a peek into the pandas source code.

Common pitfalls and solutions

Enormous datasets and memory

Just like Professor Moody always said, ‘CONSTANT VIGILANCE!’ Be watchful of memory usage when you’re juggling large data. Process in chunks or use the iterator to keep your memory in check.

Switching between arrays and lists

Switching back and forth between lists and arrays can be as disconcerting as apparitions. Keep checking your data types and avoid unnecessary conversions.

Mixed data types within a column

When columns hoard heterogeneous data types, nightmares are real. Use pd.Series.astype() to standardize your Series before conversion.