Note
Go to the end to download the full example code
Outlier Sample Detection#
This notebook provides an overview for using and understanding the Outlier Sample Detection check.
Structure:
How deepchecks detects outliers#
Outlier Sample Detection searches for outliers samples (jointly across all features) using the LoOP algorithm. The LoOP algorithm is a robust method for detecting outliers in a dataset across multiple variables by comparing the density in the area of a sample with the densities in the areas of its nearest neighbors (see the LoOp paper for further details).
LoOP relies on a distance matrix. In our implementation we use the Gower distance that averages the distances per feature between samples. For numeric features it calculates the absolute distance divided by the range of the feature and for categorical features it is an indicator for whether the values are the same (see link for further details).
Imports#
import pandas as pd
from sklearn.datasets import load_iris
from deepchecks.tabular import Dataset
from deepchecks.tabular.checks import OutlierSampleDetection
Prepare data#
iris = pd.DataFrame(load_iris().data)
iris.describe()
Add an outlier:
outlier_sample = [1, 10, 50, 100]
iris.loc[len(iris.index)] = outlier_sample
print(iris.tail())
modified_iris = Dataset(iris, cat_features=[])
0 1 2 3
146 6.3 2.5 5.0 1.9
147 6.5 3.0 5.2 2.0
148 6.2 3.4 5.4 2.3
149 5.9 3.0 5.1 1.8
150 1.0 10.0 50.0 100.0
Run the Check#
We define the nearest_neighbors_percent and the extent parameters for the LoOP algorithm.
check = OutlierSampleDetection(nearest_neighbors_percent=0.01, extent_parameter=3)
check.run(modified_iris)
Define a condition#
Now, we define a condition that enforces that the ratio of outlier samples in out dataset is below 0.001.
check = OutlierSampleDetection()
check.add_condition_outlier_ratio_less_or_equal(max_outliers_ratio=0.001, outlier_score_threshold=0.9)
check.run(modified_iris)
Total running time of the script: (0 minutes 0.113 seconds)