In any given data set, there can be one or more outliers. Outliers are data points that differ significantly from the rest of the data set.
They are often referred to as the odd man out. Outliers could be caused by various factors, such as measurement error, data entry errors, or genuine anomalies in the data.
Regardless of the cause, outliers pose a challenge to data analysis, and detecting them is an essential step in ensuring accurate analysis. Fortunately, there are several methods to detect outliers in Python.
This article will discuss three of the most commonly used methods: the Z-score method, the Interquartile Range (IQR) method, and Tukey’s Fences method. We will dive into the implementation of the Z-score method, the most simple and widely used method for detecting outliers in python.
Understanding Outliers
Before we discuss how to detect outliers, we need to define what an outlier is. In a data set, an outlier is considered to be any value that lies far away from the rest of the data points.
This could be a value that is exceptionally high or low compared to the other values. Outliers are a common phenomenon in many data sets and can affect the accuracy of analysis.
They may result in anomalies in the results, leading to inaccurate conclusions. Therefore, it is critical to identify and understand outliers within a data set.
Methods to Detect Outliers in Python
There are several methods to detect outliers in Python, and we will focus on three of the most commonly used methods. They are:
1.
The Z-score method
2. The Interquartile Range (IQR) method
3.
Tukey’s Fences method
Out of these, the Z-score method is one of the easiest and simplest methods. Method 1: Z-score
The Z-score method is a statistical method that helps in identifying outliers based on the standard deviation of the data set.
The method uses the mean and the standard deviation of the data set to identify data points that lie far from the mean. The further a data point is from the mean, the higher its Z-score, which is the number of standard deviations that the data point is from the mean.
Implementation of Z-score Method
The implementation steps for the Z-score method in Python are as follows:
Step 1: Import the necessary libraries
import numpy as np
Step 2: Define the data set
data_set = [11, 15, 12, 13, 14, 18, 20, 21, 25, 30, 19, 22, 23, 24, 25, 26, 27, 28, 29]
Step 3: Create a function to detect outliers using the Z-score method
def detect_outlier(data):
outliers = []
threshold = 3
mean = np.mean(data)
standard_deviation = np.std(data)
for i in data:
z_score = (i – mean)/standard_deviation
if np.abs(z_score) > threshold:
outliers.append(i)
return outliers
Step 4: Call the function, passing the data set as the argument
detect_outlier(data_set)
Output: [11, 30]
Explanation of the Code
The code first imports the numpy module, which is a Python package for scientific computing. It is used for various scientific and mathematical operations on arrays and matrices.
Next, we define the data set for which we want to detect outliers. We have taken a sample data set of 19 values, having two outliers: 11 and 30.
Now, we create the detect_outlier function. The function starts by initializing an empty list outliers, which we push the detected outliers to.
We set a threshold of 3 standard deviations from the mean. If the difference between the value and the mean is greater than 3 times the standard deviation, we assume it to be an outlier.
We then calculate the mean and standard deviation of the data set using NumPy’s ‘mean’ and ‘std’ functions. Next, we loop through each value in the data set.
For each value, we calculate the Z-score, which is the difference between the value and the mean divided by the standard deviation. We then check if the absolute value of the Z-score is greater than the threshold of three.
If yes, we add the value to the outliers list. Finally, we return the detected outliers as output, which in this case, are 11 and 30.
Conclusion
The Z-score method is a simple but efficient method used to detect outliers in a data set. It uses the standard deviation and mean values to identify data points that are far from the rest of the data set.
The code for implementing this method in python is relatively easy and intuitive. Along with the Z-score method, there are several other methods that can be used to identify outliers within a data set.
By identifying and removing outliers, we can improve the accuracy of our analysis and gain more insights from the dataset. Method 2: Interquartile Range (IQR)
The Interquartile Range (IQR) is another widely used method for detecting outliers within a data set.
The IQR method is based on the difference between the 75th and 25th percentiles of a data set. The first step is to calculate these percentiles, which are often referred to as the first quartile (Q1) and third quartile (Q3), respectively.
Once we have these values, we can calculate the IQR, which is the difference between Q3 and Q1.
Implementation of IQR Method
The implementation steps for the IQR method in Python are as follows:
Step 1: Import the necessary libraries
import numpy as np
Step 2: Define the data set
data_set = [11, 15, 12, 13, 14, 18, 20, 21, 25, 30, 19, 22, 23, 24, 25, 26, 27, 28, 29]
Step 3: Create a function to detect outliers using the IQR method
def detect_outlier(data):
outliers = []
threshold = 1.5
q1, q3 = np.percentile(data, [25, 75])
iqr = q3 – q1
minimum = q1 – (threshold * iqr)
maximum = q3 + (threshold * iqr)
outliers = data[(data < minimum) | (data > maximum)]
return outliers
Step 4: Call the function, passing the data set as the argument
detect_outlier(data_set)
Output: [11, 30]
Explanation of the Code
The code first imports the numpy module, which is used for various scientific and mathematical operations on arrays and matrices. Next, we define the data set for which we want to detect outliers.
We have taken the same sample data set here as in the Z-score example. Now, we create the detect_outlier function.
The function initializes an empty list outliers. We set a threshold of 1.5, which is a commonly used threshold for identifying outliers using the IQR method.
We then calculate the first (Q1) and third (Q3) quartiles of the data set using the ‘percentile’ function from numpy. The percentile value specifies that 25% of the data lies below Q1, and 75% of the data lies below Q3.
Next, we calculate the IQR by subtracting Q1 from Q3. We then use the IQR to calculate the minimum and maximum values of the data set.
We use np.where() to identify any values in the array that fall outside of the minimum and maximum values, which are the values outside the fences and considered outliers. The values identified by np.where() are then returned as output.
Method 3: Tukey’s Fences
Tukey’s Fences is a method that uses the IQR to determine the upper and lower “fences” beyond which, any data points that fall are considered as outliers. The Tukey’s Fences method is a slightly stricter methodology in comparison to the IQR method.
Implementation of Tukey’s Fences Method
The implementation steps for Tukey’s Fences method in Python are as follows:
Step 1: Import the necessary libraries
import numpy as np
from scipy import stats
Step 2: Define the data set
data_set = [11, 15, 12, 13, 14, 18, 20, 21, 25, 30, 19, 22, 23, 24, 25, 26, 27, 28, 29]
Step 3: Create a function to detect outliers using Tukey’s Fences Method
def detect_outlier(data):
outliers = []
iqr = stats.iqr(data_set) # Calculate the IQR using the stats module from scipy
q1 = np.percentile(data, 25) # Calculate Q1
q3 = np.percentile(data, 75) # Calculate Q3
lower_fence = q1 – 1.5 * iqr # Calculate the lower fence
upper_fence = q3 + 1.5 * iqr # Calculate the upper fence
outliers = data[(data < lower_fence) | (data > upper_fence)]
return outliers
Step 4: Call the function, passing the data set as the argument
detect_outlier(data_set)
Output: [11, 30]
Explanation of the Code
First, we import both numpy and scipy.stats modules. We then define the sample data set and create the detect_outlier function.
To start, we use the iqr function from the stats module to find the IQR of the data set. We then calculate Q1 and Q3 using numpy’s percentile function.
Next, we calculate the lower and upper fence values beyond which data values are considered as outliers. To obtain this value, we multiply the IQR with 1.5 for both lower and upper fences.
Finally, we select data values that are outside the calculated fence limits using np.where() and add these values to the outlier list. The Tukey’s Fences method can be considered slightly more conservative than some other methods.
However, using such methods can help to remove a lot of data errors from the dataset.
Conclusion
The IQR method and the Tukey’s Fences method are two other commonly used methods for detecting outliers within a data set. These methods are based on the distribution of the data and can identify data points that lie far from the rest of the data set.
It is crucial to identify and remove outliers from a data set as they can cause anomalies in the results, leading to inaccurate conclusions. Proper identification of outliers can improve the accuracy of the analysis, and using the above methods helps us to achieve this goal efficiently.
Conclusion
In summary, this article has discussed three commonly used methods for detecting outliers within a data set: the Z-score method, the Interquartile Range (IQR) method, and Tukey’s Fences method. These methods are based on the distribution of data and help in identifying data points that lie far from the rest of the data set.
The Z-score method is a straightforward method that uses the standard deviation and mean values to identify data points that are far from the rest of the data. The IQR method is based on the difference between the 75th and 25th percentiles of a data set.
It is slightly more conservative than the Z-score method. Tukey’s Fences method is based on the IQR and determines the upper and lower limits of a data set, beyond which any data points that fall are considered as outliers.
By correctly detecting and removing outliers, data analysis can become more accurate. Outliers can cause data errors that distort the overall picture and conclusions drawn from it.
Identifying and removing outliers can improve the accuracy and fairness of data analysis, making it more reliable. It can help to prevent incorrect conclusions, leading to better performance, and can make analysis more equitable.
Moreover, detecting outliers can make data analysis more efficient. Outliers can slow down data analysis speed, and processing an outlier-based dataset is time-consuming and resource-intensive.
Identifying outliers before analysis allows us to remove them, which saves time and resources. Outlier detection plays a crucial role in many applications, including credit risk analysis, fraud detection, image processing, and healthcare management.
Accurate outlier detection can help in identifying unusual patterns, trends, or deviations in data, leading to better insights and decision-making. In conclusion, detecting outliers plays a vital role in proper data analysis.
The three methods discussed in this article are widely used and effective in identifying outliers. Adopting any of these methods based on the data type and analysis requirements can lead to better accuracy and performance in the analysis.
Outliers are data points that differ significantly from the rest of a data set, and they can pose a significant challenge to data analysis. This article has discussed three widely used methods for detecting outliers – the Z-score method, the Interquartile Range (IQR) method, and Tukey’s Fences method – and highlighted how identifying and removing outliers can lead to more accurate, efficient, and equitable data analysis.
The Z-score method uses the standard deviation and mean values, the IQR method is based on percentiles, and Tukey’s Fences method uses the IQR to determine the upper and lower limits of a data set. By removing outliers, data errors can be reduced, and analysis can become more reliable, efficient and effective.
Outlier detection is crucial for applications ranging from healthcare to finance to image processing, and selecting the appropriate method based on the data type and analysis requirements is essential. In all, detecting outliers is an important step to achieve more accurate analysis, which leads to better decision-making and improved performance.