# Mastering Pandas: Essential Techniques for Data Analysis and Manipulation

Are you a data analyst or an aspiring data analyst looking to work with data using the powerful Python library, Pandas? If so, you’re in the right place.

In this article, we’ll explore two essential topics in Pandas that will make data analysis and manipulation a breeze – finding unique values and creating a Pandas DataFrame.

## Finding Unique Values in Pandas and Ignoring NaN Values

When working with large datasets, it’s often necessary to identify unique values to gain insights into the data. However, missing or NaN values can often pose a challenge that requires specialized solutions.

In this section, we’ll explore two examples of how to find unique values in Pandas while ignoring NaN values.

## Custom Function for Finding Unique Values

Pandas provides a unique() method that returns the unique values in a Pandas series or column. However, this method does not ignore NaN values.

“`

## import pandas as pd

data = {‘name’: [‘Alex’, ‘Ben’, ‘Chris’, ‘Dave’, ‘Ella’, ‘Fiona’, ‘George’, ‘Hannah’],

age’: [22, 19, 21, 22, 18, 23, 21, 22],

‘score’: [85, 78, 92, 90, 75, 96, None, None]}

df = pd.DataFrame(data)

“`

We can see that the ‘score’ column has NaN values, as indicated by the ‘None’ values in the dataframe. To find unique values in the ‘score’ column while ignoring NaN values, we can use a custom function as follows:

“`

def unique_vals_ignore_na(col):

vals= col.unique().tolist()

if pd.isna(vals):

vals.pop(0)

return vals

“`

This function first finds the unique values in the column using the unique() method and stores them in a list.

The function then checks if the first value in the list is NaN using the pd.isna() function and removes it if present using the list.pop() method. Finally, the function returns the resulting list of unique values.

Example 1: Finding Unique Values in Pandas Column and Ignoring NaN Values

To find unique values in the ‘score’ column of the dataframe while ignoring NaN values using the custom function, we can do the following:

“`

unique_scores = unique_vals_ignore_na(df[‘score’])

## print(unique_scores)

“`

This should output: `[85.0, 78.0, 92.0, 90.0, 75.0, 96.0]`

Example 2: Finding Unique Values in Pandas Groupby and Ignoring NaN Values

To find unique values in a pandas groupby object while ignoring NaN values using the custom function, we can use the following code snippet:

“`

grouped = df.groupby(‘

age’)[‘score’].apply(lambda x: unique_vals_ignore_na(x))

“`

“`

## age

18 [75.0]

19 [78.0]

21 [92.0, 96.0]

22 [85.0, 90.0]

23 [nan]

Name: score, dtype: object

“`

## Creating a Pandas DataFrame

Creating a Pandas DataFrame is one of the first steps in working with Pandas for data analysis. We create DataFrames as a way of representing and manipulating data in tabular form.

In this section, we’ll explore how to create a Pandas DataFrame and view its contents.

## Importing Required Libraries

Before creating a DataFrame, we may need to import the required libraries, namely Pandas and NumPy. Here’s a code snippet that shows how to import these libraries:

“`

“`

## Creating DataFrame

To create a Pandas DataFrame, we can use various methods. One of the easiest ways is to use a dictionary to define our data.

Here’s an example:

“`

data = {‘name’: [‘Alex’, ‘Ben’, ‘Chris’, ‘Dave’, ‘Ella’, ‘Fiona’, ‘George’, ‘Hannah’],

age’: [22, 19, 21, 22, 18, 23, 21, 22],

‘score’: [85, 78, 92, 90, 75, 96, None, None]}

df = pd.DataFrame(data)

“`

In this example, we define a dictionary data and use the pd.DataFrame() function to create a Pandas DataFrame from the dictionary. The resulting DataFrame will have columns as specified in the dictionary’s keys and values as specified in the dictionary’s values.

## Viewing DataFrame

To view the contents of a DataFrame, we can use the head() and tail() methods. The head() method returns the first n rows of the DataFrame, while the tail() method returns the last n rows of the DataFrame.

“`

“`

“`

name

## age score

0 Alex 22 85.0

1 Ben 19 78.0

2 Chris 21 92.0

3 Dave 22 90.0

4 Ella 18 75.0

“`

We’ve covered two essential topics in Pandas that will make your journey into data analysis much smoother. By learning how to find unique values while ignoring NaN values and how to create a Pandas DataFrame, you’re well on your way to manipulating and analyzing data with ease.

With practice, these concepts will become second nature to you, enabling you to unlock insights and make data-driven decisions.

## 3) Displaying Basic Information about a Pandas DataFrame

When working with a Pandas DataFrame, it’s often useful to display basic information about the data to obtain an understanding of its structure and content. In this section, we’ll explore how to display column names, data types, and shape of the DataFrame.

## Displaying Column Names

To display the column names of a Pandas DataFrame, we can use the columns attribute. Here’s an example:

“`

## import pandas as pd

data = {‘name’: [‘Alex’, ‘Ben’, ‘Chris’, ‘Dave’, ‘Ella’, ‘Fiona’, ‘George’, ‘Hannah’],

age’: [22, 19, 21, 22, 18, 23, 21, 22],

‘score’: [85, 78, 92, 90, 75, 96, None, None]}

df = pd.DataFrame(data)

print(df.columns)

“`

This should output: `Index([‘name’, ‘

age’, ‘score’], dtype=’object’)`

The columns attribute returns a Pandas Index object that contains the column names.

## Displaying Data Types

To display the data types of each column in a Pandas DataFrame, we can use the dtypes attribute. Here’s an example:

“`

print(df.dtypes)

“`

“`

## score float64

dtype: object

“`

The dtypes attribute returns a Pandas series that contains the data types of each column.

## Displaying Shape of DataFrame

To display the shape or dimensions of a Pandas DataFrame, we can use the shape attribute. Here’s an example:

“`

print(df.shape)

“`

This should output: `(8, 3)`

The shape attribute returns a tuple that contains the number of rows and columns in the DataFrame.

## 4) Accessing Data in a Pandas DataFrame

Accessing data in a Pandas DataFrame is a critical part of data analysis. In this section, we’ll explore how to access rows and columns using indexing, iloc, and loc.

## Accessing Rows and Columns Using Indexing

To access a specific row or column in a Pandas DataFrame using indexing, we can use the iloc attribute. Here’s an example:

“`

## import pandas as pd

data = {‘name’: [‘Alex’, ‘Ben’, ‘Chris’, ‘Dave’, ‘Ella’, ‘Fiona’, ‘George’, ‘Hannah’],

age’: [22, 19, 21, 22, 18, 23, 21, 22],

‘score’: [85, 78, 92, 90, 75, 96, None, None]}

df = pd.DataFrame(data)

# accessing a specific column by index

print(df.iloc[:, 1])

“`

“`

## 7 22

Name:

age, dtype: int64

“`

The iloc attribute allows us to access data using the numerical index of the row or column. In this example, we access the second column (index 1) using the syntax df.iloc[:, 1], which means accessing all rows and the second column.

Similarly, we can access a specific row using the index of its position as shown below:

“`

# accessing a specific row by index

print(df.iloc[2, :])

“`

“`

## age 21

score 92.0

Name: 2, dtype: object

“`

## Accessing Data Using iloc

To access specific rows and columns of a Pandas DataFrame using iloc, we can use the following syntax:

“`

df.iloc[row_start:row_end, col_start:col_end]

“`

This syntax allows us to specify a range of rows and columns to access. Here’s an example:

“`

# accessing a range of rows and columns using iloc

print(df.iloc[2:5, 0:2])

“`

“`

name

## 4 Ella 18

“`

In this example, we accessed rows 2 to 4 and columns 0 to 1, which returns all rows between the 2nd and 4th row and the first and second columns.

## Accessing Data Using loc

The loc attribute allows us to access data in a Pandas DataFrame using labels or names instead of numerical indexes. Here’s an example:

“`

# accessing a specific row using loc

print(df.loc)

“`

“`

## score 90

Name: 3, dtype: object

“`

In this example, we accessed the row labeled 3. We can also access a specific column using loc as shown below:

“`

# accessing a specific column using loc

print(df.loc[:, ‘score’])

“`

“`

0 85.0

1 78.0

2 92.0

3 90.0

4 75.0

5 96.0

## 7 NaN

Name: score, dtype: float64

“`

In this example, we accessed the ‘score’ column using the loc attribute. In conclusion, accessing and displaying basic information in a Pandas DataFrame is essential in data analysis.

Understanding how to access data using indexing, iloc, and loc, as well as displaying column names, data types, and shape, will enable you to manipulate and analyze data with ease. With practice, these concepts will become second nature, allowing you to unlock insights and make data-driven decisions.

## 5) Modifying Data in a Pandas DataFrame

Modifying data in a Pandas DataFrame is an essential part of data analysis. In this section, we’ll explore how to modify data in specific columns, add a new column to a DataFrame, and drop rows with NaN values.

## Modifying Data in Specific Columns

To modify data in specific columns of a Pandas DataFrame, we can use the following syntax:

“`

df[‘column_name’] = new_values

“`

Here, we replace ‘column_name’ with the name of the column we wish to modify and ‘new_values’ with the new values that we want to assign to the column. Here’s an example:

“`

## import pandas as pd

data = {‘name’: [‘Alex’, ‘Ben’, ‘Chris’, ‘Dave’, ‘Ella’, ‘Fiona’, ‘George’, ‘Hannah’],

age’: [22, 19, 21, 22, 18, 23, 21, 22],

‘score’: [85, 78, 92, 90, 75, 96, None, None]}

df = pd.DataFrame(data)

# modifying data in the ‘score’ column

df[‘score’] = [85, 78, 92, 90, 75, 96, 87, 81]

“`

“`

name

## 7 Hannah 22 81

“`

In this example, we modified the data in the ‘score’ column by assigning new values to the column.

## Adding a New Column to a DataFrame

Adding a new column to a Pandas DataFrame is easy. We can use the following syntax:

“`

df[‘new_column_name’] = new_column_data

“`

Here, we replace ‘new_column_name’ with the name of the column to be added and ‘new_column_data’ with the values that we want to assign to the new column.

Here’s an example:

“`

# adding a new column to the DataFrame

df[‘grade’] = [‘A’, ‘C’, ‘A’, ‘A’, ‘C’, ‘A’, ‘B’, ‘B’]

“`

“`

name

## 7 Hannah 22 81 B

“`

In this example, we added a new column ‘grade’ to the DataFrame and assigned the appropriate values.

## Dropping Rows with NaN Values

Sometimes, we may wish to drop rows with NaN values from a Pandas DataFrame. To do this, we can use the following syntax:

“`

df.dropna(axis=0, inplace=True)

“`

Here, the dropna() method drops all rows that have at least one NaN value.

The axis parameter specifies that we want to drop rows (axis=0) with NaN values. The inplace parameter specifies that the DataFrame should be modified directly.

## 6) Filtering Data in a Pandas DataFrame

Filtering data in a Pandas DataFrame is essential in data analysis. In this section, we’ll explore how to filter data based on values in a specific column, chain multiple filters together, and use lambda functions in filtering.

## Filtering Data Based on Values in a Column

To filter a Pandas DataFrame based on values in a specific column, we can use the following syntax:

“`

df[df[‘column_name’] == value]

“`

Here, we replace ‘column_name’ with the name of the column we wish to filter and ‘value’ with the value that we want to filter by. Here’s an example:

“`

## import pandas as pd

data = {‘name’: [‘Alex’, ‘Ben’, ‘Chris’, ‘Dave’, ‘Ella’, ‘Fiona’, ‘George’, ‘Hannah’],

age’: [22, 19, 21, 22, 18, 23, 21, 22],

‘score’: [85, 78, 92, 90, 75, 96,