Adventures in Machine Learning

Mastering Data Manipulation and Analysis with Pandas

Data analysis is a critical aspect of any business or research project. Tabular data is often the norm in data analysis, and managing tabular data requires the use of specialized tools.

Pandas is currently one of the most popular Python libraries for data manipulation and analysis. In this article, we will cover two critical aspects of pandas – dataframe manipulation using pandas functions and using the assign() method to add new columns to dataframes.

Assign() Method

The assign() method is a powerful and convenient way to add new columns to a dataframe based on existing columns or new variables. The assign() method returns a new dataframe with the added columns while leaving the original dataframe unchanged.

The syntax for the assign() method is simple:


Here, dataframe refers to the dataframe you want to modify, new_column_name is the name of the new column, and new_values are the values to be assigned to the new column.

Assigning a new variable to a dataframe is straightforward.

All you need to do is define the new variable and use the assign() method to add it to the dataframe. For example, suppose you have a dataframe with two columns, “height” and “weight.” You can create a new variable, “BMI,” by dividing weight by height squared and then assign the new variable to the dataframe as shown below:

BMI = data['weight'] / (data['height'] ** 2)
data = data.assign(BMI=BMI)

It’s also possible to assign multiple new variables to a dataframe.

For example, suppose you have a dataframe with two columns, “quantity” and “price.” You can add new columns “total” and “discount” as follows:

data = data.assign(total=data['quantity'] * data['price'], discount=0.1 * data['total'])

Pandas Functions

Pandas provides a variety of functions that simplify common data manipulation tasks. These functions are efficient and easy to use and are essential tools for any data analyst or scientist.

1. Merge() Function

The merge() function is used to combine two dataframes into a single dataframe based on one or more common columns. The merge() function uses a SQL-like syntax, allowing users to perform different types of joins, including inner, left, right, and outer joins.

Let’s look at an example of how to use the merge() function:

merged_data = pd.merge(data1, data2, on='column_name', how='inner')

In this example, data1 and data2 are the dataframes to be merged, column_name is the column shared by both dataframes, and how is the type of merge to be executed. Depending on the type of join selected, there may be missing values in the resulting dataframe.

2. Pivot_table() Function

The pivot_table() function is used to reorganize a dataframe, allowing users to group data by one or more columns and aggregate the data based on those columns. The pivot_table() function is particularly useful when analyzing large datasets.

Here’s an example of how to use the pivot_table() function:

pivot_data = data.pivot_table(index=['column1', 'column2'], values=['value1', 'value2'], aggfunc=['sum', 'mean'])

Here, column1 and column2 refer to the columns to group the data by while value1 and value2 are the columns to aggregate. The aggfunc parameter specifies the aggregation functions to be used on the data, which, in this case, are sum and mean.


Pandas is a versatile library that provides extensive support for data manipulation and analysis tasks. In this article, we have discussed the assign() method, which is used to add new columns to a dataframe and pandas functions such as merge() and pivot_table(), which are used to manipulate data.

As data analysis becomes more critical, mastering these tools is crucial for any data scientist. Pandas is a versatile library for data manipulation and analysis, known for its ease of use and powerful capabilities.

Filtering and Selecting Data

Filtering and selecting data are crucial in data analysis tasks, and pandas provides simple and efficient ways to achieve these tasks. The basic syntax for filtering data in pandas is simple.


Here, dataframe is the dataframe you would like to filter, and the condition is a boolean expression. The loc[] method selects rows based on the provided condition.

For example, suppose you have a dataframe with columns “Age” and “Gender.” If you want to filter the rows where “Age” is greater than 25, you can use the loc[] method as follows:

filtered_data = data.loc[data['Age'] > 25]

Another way to filter data in pandas is by using boolean indexing. Boolean indexing uses a boolean expression to select data from a dataframe.

In boolean indexing, the condition is used to create a Boolean series, which is then used to filter the data by selecting the rows containing True values. The syntax for boolean indexing is:


For example, suppose you have a dataframe with columns “Age” and “Gender.” If you want to select rows where the “Age” is greater than 25 and the “Gender” is Male, you can use boolean indexing as follows:

selected_data = data[(data['Age'] > 25) & (data['Gender'] == 'Male')]

Grouping and Aggregating Data

Grouping and aggregating data are valuable in data analysis, and pandas has several powerful tools for these tasks. The basic syntax for grouping and aggregating data in pandas is:


Here, the groupby() method first groups the data by the column specified in column_name.

The agg() function then performs the specified aggregation function on the grouped data. Let’s look at an example.

Suppose we have a dataframe that contains information on students’ test scores and their respective subjects.

      Name     Subject  Score

0     John     Physics     90
1     John  Mathematics    100
2     John    Chemistry     85
3  Suzanne    chemistry     92
4  Suzanne  Mathematics     95
5  Suzanne     Physics     88

If we want to group the data by subject and calculate the mean score for each subject, we can use the groupby() method and the mean() function as follows:

subject_scores = data.groupby('Subject').agg('mean')

The resulting dataframe would look like:


Chemistry    88.5
Mathematics  97.5
Physics      89.0

Additionally, the agg() method can perform multiple aggregation functions simultaneously on the grouped data. Here’s an example:

subject_scores = data.groupby('Subject').agg(['max', 'min', 'mean'])

The resulting dataframe would look like:

              max min  mean

Chemistry      92  85  88.5
Mathematics   100  95  97.5
Physics        90  88  89.0


Pandas offers powerful tools for filtering, selecting, grouping and aggregating data in data analysis. This article has covered some of the basic, yet critical aspects of these tools.

As you apply these methods in your data analysis projects, it’s essential to stay vigilant as working with large datasets may slow down the performance of these methods, thereby potentially leading to memory overflow errors. Nonetheless, pandas provides an efficient data manipulation solution.

In conclusion, Pandas provides essential tools for data manipulation and analysis, including filtering and selecting data, grouping, and aggregating. The article highlighted some of the basic syntax and examples for using these tools, such as filtering data using the loc[] method or boolean indexing, or grouping data using the groupby() method and aggregating data with the agg() method.

These tools are important in any data analysis project and help to ensure efficient and accurate data manipulation. As data analysis becomes increasingly critical to businesses and organizations, achieving proficiency with Pandas will be necessary.

With these tools and their capabilities, users can streamline their data analysis and deliver more effective results.

Popular Posts