Dropping infinite values from dataframes in pandas?
Let's kick out those arithmetic outcasts, infinity and negative infinity, from your DataFrame in just one code line:
Pandas .replace() method swaps both types of infinities (np.inf and -np.inf) with the newbie NaN, who is then shown the exit with .dropna().
Want to temporarily treat infinities as NaN within a particular context? Then you'll need:
If banishing infinities forever is your plan, use:
To keep only finite values across your DataFrame:
If you're after infinities in certain columns:
Data integrity is crucial. Make sure you're not sending any valuable data into the exile of dropped values unawares.
Silence Of The Infinities
Say you have some calculations in your DataFrame that result in positive or negative infinity. They’re not harmful per se, but they could skew your analysis or clutter up your data visualization.
Handling infinite with NaN
Choosing to handle these infinite values like they are NaN can help simplify your data operations. By converting these “infinites" into NaNs, you can easily identify and eventually remove them.
Scenario specifics
Focusing on a particular column to get rid of infinite values?
Is there a column that should only retain finite values?
Got a dataframe that needs infinite replacing across multiple columns based on dtype?
On older versions of Pandas, you'll need use_inf_as_null instead of use_inf_as_na. Keep an eye on your versions!
Wrangling those infinite values
Sometimes non-numeric data types or masked values can trip up this method. Thus, it’s beneficial to conduct preliminary data exploration to understand what you’re dealing with. And remember to tailor the solution to your data.
Another approach is to create a cleaning function for your data, which handles infinities alongside other data cleaning steps.
Going off-script and writing your own function to identify and handle specific data issues will let you debug unexpected results in a jiffy! Consider integrating this within a try/except block for maximum efficiency.
Was this article helpful?