Finding Columns with NaN values in Pandas DataFrame
Data analysis often requires inspecting datasets and identifying missing or invalid values before proceeding with data cleaning. One way to identify these values is by finding columns with NaN values in a pandas DataFrame.
In this article, we will explore four approaches on how to do this.
1) Finding Columns with NaN values in Pandas DataFrame
The isna() method is a pandas method that checks if a DataFrame has NaN values.
It returns a boolean value, where True means that a column has a NaN value. We can use this method to create a mask to show which columns have NaN values.
First, let us import pandas and create a sample DataFrame with NaN values:
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, 21, 19],
'Salary': [50000, 52000, 45000, None]}
df = pd.DataFrame(data)
We can now use the isna() method to check for NaN values in the dataframe:
mask = df.isna().any()
print(mask)
Output:
Name False
Age False
Salary True
dtype: bool
The resulting output shows that the ‘Salary’ column has NaN values.
2) Using isnull() method to find columns with NaN values
The isnull() method is similar to isna() in that it also returns a boolean value to indicate missing or invalid values.
We can use this method to create the same mask as the isna() method to identify which columns have NaN values.
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, 21, 19],
'Salary': [50000, None, 45000, 55000]}
df = pd.DataFrame(data)
mask = df.isnull().any()
print(mask)
Output:
Name False
Age True
Salary True
dtype: bool
The output shows that both the ‘Age’ and ‘Salary’ columns have NaN values.
3) Using isna() method to select columns with NaN values
Another way to identify columns with NaN values is to select only the columns that have NaN values.
We can create a sub-dataframe by selecting only these columns and inspect the data further.
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, 21, 19],
'Salary': [50000, 55000, None, 45000]}
df = pd.DataFrame(data)
selected_columns = df.loc[:, df.isna().any()]
print(selected_columns)
Output:
Salary
0 50000.0
1 55000.0
2 NaN
3 45000.0
The resulting output shows only the ‘Salary’ column, which has NaN values.
4) Using isnull() method to select columns with NaN values
Similar to the previous approach, we can also use the isnull() method to select columns with NaN values.
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, 21, 19],
'Salary': [50000, None, 45000, 55000]}
df = pd.DataFrame(data)
selected_columns = df.loc[:, df.isnull().any()]
print(selected_columns)
Output:
Age Salary
0 25 50000
1 33 None
2 21 45000
3 19 55000
The resulting output shows both the ‘Age’ and ‘Salary’ columns, which have NaN values.
Creating a DataFrame
Creating a DataFrame is an essential step in data analysis. A DataFrame is a table-like data structure that contains rows and columns, where each column can contain different data.
We can create a DataFrame using the pandas DataFrame constructor. Let us see an example:
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, 21, 19],
'Salary': [50000, 52000, 45000, None]}
df = pd.DataFrame(data, columns=['Name', 'Age', 'Salary'])
print(df)
Output:
Name Age Salary
0 John 25 50000.0
1 Peter 33 52000.0
2 Mary 21 45000.0
3 Anna 19 NaN
In this example, we created a DataFrame with three columns: ‘Name’, ‘Age’, and ‘Salary’. We also specified the column order using the ‘columns’ parameter in the pandas DataFrame constructor.
We then printed the resulting DataFrame, which includes the specified data and NaN values under the ‘Salary’ column.
Conclusion
In this article, we learned how to identify columns with NaN values in a pandas DataFrame using four different approaches. We also discussed how to create a DataFrame using the pandas DataFrame constructor.
With this knowledge, we can continue with data analysis by cleaning and filling in missing values to avoid errors in our calculations.
3) Finding all Columns with NaN Values in Pandas DataFrame
During data analysis, we often come across datasets that have missing or invalid values. To ensure accurate data analysis, we need to find all the columns that have NaN values.
In this section, we will talk about two methods to find all columns with NaN values in a pandas DataFrame.
Step 1: Creating the DataFrame
Before we dive into the methods, let’s first create a sample DataFrame with NaN values:
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, None, 19],
'Salary': [50000, 52000, None, 55000],
'Education': [None, "Bachelor's", "Bachelor's", "High School"]}
df = pd.DataFrame(data)
print(df)
Output:
Name Age Salary Education
0 John 25.0 50000.0 None
1 Peter 33.0 52000.0 Bachelor's
2 Mary NaN NaN Bachelor's
3 Anna 19.0 55000.0 High School
In this example, we have created a DataFrame with four columns: ‘Name’, ‘Age’, ‘Salary’, and ‘Education’. We have intentionally put NaN values in the ‘Age’, ‘Salary’, and ‘Education’ columns.
Step 2: Using isna() or isnull() to find all columns with NaN values
To find all columns with NaN values, we can use either the isna() method or the isnull() method. Both these methods generate a boolean DataFrame where True represents a NaN value.
We can then use the any() method to see if any column has the value True, which means it has a NaN value.
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, None, 19],
'Salary': [50000, 52000, None, 55000],
'Education': [None, "Bachelor's", "Bachelor's", "High School"]}
df = pd.DataFrame(data)
# Using isna() method
mask1 = df.isna().any()
print(mask1)
# Using isnull() method
mask2 = df.isnull().any()
print(mask2)
Output:
Name False
Age True
Salary True
Education True
dtype: bool
Name False
Age True
Salary True
Education True
dtype: bool
As we can see from the output, all three columns that have NaN values are identified as True in the boolean DataFrame.
4) Selecting all Columns with NaN Values in Pandas DataFrame
Now that we know how to find all columns with NaN values, we may want to select only these columns to inspect them further. In this section, we will talk about two methods to select all columns with NaN values in a pandas DataFrame.
Approach 1: Using isna() method to select all columns with NaN values
One way to select all columns with NaN values is to use the isna() method to create a boolean mask. We can then use this mask and the loc[] method to select the desired columns.
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, None, 19],
'Salary': [50000, 52000, None, 55000],
'Education': [None, "Bachelor's", "Bachelor's", "High School"]}
df = pd.DataFrame(data)
# Identify columns with NaN values
mask = df.isna().any()
# Select only columns with NaN values
selected_columns = df.loc[:, mask]
print(selected_columns)
Output:
Age Salary Education
0 25.0 50000.0 None
1 33.0 52000.0 Bachelor's
2 NaN NaN Bachelor's
3 19.0 55000.0 High School
As we can see, the output only shows the columns that have NaN values, namely ‘Age’, ‘Salary’, and ‘Education’.
Approach 2: Using isnull() method to select all columns with NaN values
We can also use the isnull() method to select all columns with NaN values.
We can create a boolean mask using this method, then use it with the columns method to get the desired columns.
import pandas as pd
data = {'Name': ['John', 'Peter', 'Mary', 'Anna'],
'Age': [25, 33, None, 19],
'Salary': [50000, 52000, None, 55000],
'Education': [None, "Bachelor's", "Bachelor's", "High School"]}
df = pd.DataFrame(data)
# Identify columns with NaN values
mask = df.isnull().any()
# Select only columns with NaN values
selected_columns = df[df.columns[df.isnull().any()]]
print(selected_columns)
Output:
Age Salary Education
0 25.0 50000.0 None
1 33.0 52000.0 Bachelor's
2 NaN NaN Bachelor's
3 19.0 55000.0 High School
The output is the same as in approach 1, with only the columns that have NaN values.
Conclusion
In this article expansion, we have learned two methods to find and select columns with NaN values in a pandas DataFrame. We can use the isna() or isnull() methods to generate a boolean mask to identify the desired columns.
We can then use this mask to select only the columns with NaN values. This knowledge is essential in data analysis, where dealing with missing values is crucial to ensure accurate results.
In this article, we explored various methods to identify and select columns with NaN values in a pandas DataFrame. We learned that finding and selecting these columns is essential in data analysis, as missing values can significantly impact the accuracy of our results.
The isna() and isnull() methods help to identify columns with NaN values, while the loc[] and columns() methods allow us to select only these columns for further analysis. By understanding these methods, we can better clean and treat missing values in our data sets, improving the accuracy and reliability of our data analysis.