Outlier Sample Detection#

This notebook provides an overview for using and understanding the Outlier Sample Detection check.

Structure:

How deepchecks detects outliers#

Outlier Sample Detection searches for outliers samples (jointly across all features) using the LoOP algorithm. The LoOP algorithm is a robust method for detecting outliers in a dataset across multiple variables by comparing the density in the area of a sample with the densities in the areas of its nearest neighbors (see the LoOp paper for further details).

LoOP relies on a distance matrix. In our implementation we use the Gower distance that averages the distances per feature between samples. For numeric features it calculates the absolute distance divided by the range of the feature and for categorical features it is an indicator for whether the values are the same (see link for further details).

Imports#

import pandas as pd
from sklearn.datasets import load_iris

from deepchecks.tabular import Dataset
from deepchecks.tabular.checks import OutlierSampleDetection

Prepare data#

iris = pd.DataFrame(load_iris().data)
iris.describe()
0 1 2 3
count 150.000000 150.000000 150.000000 150.000000
mean 5.843333 3.057333 3.758000 1.199333
std 0.828066 0.435866 1.765298 0.762238
min 4.300000 2.000000 1.000000 0.100000
25% 5.100000 2.800000 1.600000 0.300000
50% 5.800000 3.000000 4.350000 1.300000
75% 6.400000 3.300000 5.100000 1.800000
max 7.900000 4.400000 6.900000 2.500000


Add an outlier:

outlier_sample = [1, 10, 50, 100]
iris.loc[len(iris.index)] = outlier_sample
print(iris.tail())
modified_iris = Dataset(iris, cat_features=[])
       0     1     2      3
146  6.3   2.5   5.0    1.9
147  6.5   3.0   5.2    2.0
148  6.2   3.4   5.4    2.3
149  5.9   3.0   5.1    1.8
150  1.0  10.0  50.0  100.0

Run the Check#

We define the nearest_neighbors_percent and the extent parameters for the LoOP algorithm.

check = OutlierSampleDetection(nearest_neighbors_percent=0.01, extent_parameter=3)
check.run(modified_iris)
Outlier Sample Detection


Define a condition#

Now, we define a condition that enforces that the ratio of outlier samples in out dataset is below 0.001.

check = OutlierSampleDetection()
check.add_condition_outlier_ratio_less_or_equal(max_outliers_ratio=0.001, outlier_score_threshold=0.9)
check.run(modified_iris)
Outlier Sample Detection


Total running time of the script: (0 minutes 0.076 seconds)

Gallery generated by Sphinx-Gallery