Adventures in Machine Learning

Mastering Categorical Data Conversion with Pandas factorize()

Understanding and Implementing the Pandas factorize() Function

Have you ever had to deal with categorical data in Python and found yourself struggling to convert it to numerical data? One way to handle this is by using the Pandas factorize() function.

This function returns an enumerated array and a unique set of values, which makes it easier to work with categorical data. In this article, we will explore the syntax, use cases, and implementation of the factorize() function in Pandas.

Syntax of the factorize() Function

The Pandas factorize() function takes an input array and returns a tuple containing two arrays. The first array is an integer code representing each unique value in the input array.

The second array is a list of the unique values. Here is the syntax for the factorize() function:

factorize(self, values, sort=False, na_sentinel=-1)

The input parameters are:

  • values – the input array of labels
  • sort – a boolean option that sorts the unique values before returning
  • na_sentinel – a sentinel value to mark missing values

Use Cases for the factorize() Function

One common use case for the factorize() function is to convert categorical data into numerical data. This is important because some machine learning algorithms require numerical data as input.

Another use case is to identify unique values in a dataset. Here are some example input values:

languages = ['English', 'Spanish', 'French', 'German', 'English', 'Spanish', 'Italian']

To factorize this list, we can simply pass it as the values parameter to the factorize() function:

codes, uniques = pd.factorize(languages)

The output of the factorize() function will be:

codes: [0 1 2 3 0 1 4]
uniques: ['English', 'Spanish', 'French', 'German', 'Italian']

Deploying Using Default Setting

By default, the factorize() function uses the first occurrence of a value as the code, but you can also specify the code yourself. Let’s take a look at how the function behaves with the default settings on a set of values:

colors = ['red', 'blue', 'green', 'red', 'green', 'yellow']
codes, uniques = pd.factorize(colors)

The output of the factorize() function will be:

codes: [0 1 2 0 2 3]
uniques: ['red', 'blue', 'green', 'yellow']

Extracting the Code & Unique Part of the Result

As we have already mentioned, the factorize() function returns a tuple of two arrays – the code and the unique part of the result.

To extract these arrays, we simply assign the output of the function to two variables:

codes, uniques = pd.factorize(colors)

Now, we can print out the code and unique arrays as separate arrays:

print(codes) # Output: [0 1 2 0 2 3]
print(uniques) # Output: ['red', 'blue', 'green', 'yellow']

Sorting the Result

Sometimes, it’s helpful to have the output of the factorize() function sorted according to the codes or the unique values. To do this, we can use the sort option.

Here is an example:

codes, uniques = pd.factorize(colors, sort=True)

Now, when we print out the codes and unique arrays, they will be sorted according to the unique values:

print(codes) # Output: [0 1 2 0 2 3]
print(uniques) # Output: ['blue', 'green', 'red', 'yellow']

Factorizing None values

If you have missing or None values in your dataset, you can specify a sentinel value to mark them. By default, the factorize() function marks missing values with -1.

Here is an example:

sizes = ['small', 'medium', None, 'large', 'medium']
codes, uniques = pd.factorize(sizes, na_sentinel=-999)
print(codes) # Output: [0 1 -999 2 1]

Implementing the Pandas factorize() Function

Now that we have gone over the basics of the factorize() function, let’s take a look at how to implement it in a DataFrame.

Implementing in a DataFrame

The factorize() function can be used on a Pandas DataFrame to convert categorical columns to numerical columns. Here is an example:

import pandas as pd
data = {'countries': ['USA', 'Japan', 'Russia', 'USA', 'Russia'],
        'cities': ['New York', 'Tokyo', 'Moscow', 'Los Angeles', 'St. Petersburg']}
df = pd.DataFrame(data)
df['countries_num'], _ = pd.factorize(df['countries'])
df['cities_num'], _ = pd.factorize(df['cities'])
print(df)

The output of this code will be:

    countries         cities  countries_num  cities_num
0        USA      New York              0           0
1      Japan         Tokyo              1           1
2     Russia        Moscow              2           2
3        USA   Los Angeles              0           3
4     Russia  St. Petersburg          2           4

Handling Missing Values

When working with DataFrames, it is common to have missing values in the dataset. The factorize() function provides two ways to handle missing values: dropna() and fillna().

Dropna() will remove any rows that contain missing values:

df.dropna(inplace=True)

Fillna() will replace missing values with a specified value:

df.fillna(value=-1, inplace=True)

Combining factorize() with groupby()

Another powerful feature of the factorize() function is its ability to be combined with the groupby() function. Here is an example:

import pandas as pd
data = {'countries': ['USA', 'Japan', 'Russia', 'USA', 'Russia'],
        'cities': ['New York', 'Tokyo', 'Moscow', 'Los Angeles', 'St. Petersburg'],
        'revenue': [100, 200, 150, 120, 180]}
df = pd.DataFrame(data)
df['countries_num'], _ = pd.factorize(df['countries'])
df['cities_num'], _ = pd.factorize(df['cities'])
grouped = df.groupby(['countries_num', 'cities_num']).sum()
print(grouped)

The output of this code will be:

                      revenue

countries_num cities_num        
0             0            100
              3            120
1             1            200
2             2            150
              4            180

Conclusion:

The Pandas factorize() function is a powerful tool in the world of data analysis. It allows you to convert categorical variables to numerical variables, identify unique values in a dataset, and combine factorized data with groupby() functions.

The function is easy to use and provides several options to customize its behavior. Whether you are working with a small dataset or a large one, the factorize() function is a great way to streamline your analysis and extract meaningful insights.

In conclusion, the Pandas factorize() function is a useful tool for data analysis that enables you to convert categorical data into numerical data, identify unique values in a dataset, and combine factorized data with groupby() functions. It is a versatile function that can be used on a DataFrame, and it provides several options to customize its behavior.

The function’s ability to handle missing data and to sort output based on unique values or codes makes it a valuable tool for data analysts and scientists. By using it, you can streamline your data analysis workflow and extract meaningful insights.

Popular Posts