• API Reference - OutlierSampleDetection

Outlier Sample Detection

This notebook provides an overview for using and understanding the Outlier Sample Detection check.

Structure:

• How deepchecks detects outliers

• Prepare data

• Run the check

• Define a condition

How deepchecks detects outliers

Outlier Sample Detection searches for outliers samples (jointly across all features) using the LoOP algorithm. The LoOP algorithm is a robust method for detecting outliers in a dataset across multiple variables by comparing the density in the area of a sample with the densities in the areas of its nearest neighbors (see the LoOp paper for further details).

LoOP relies on a distance matrix. In our implementation we use the Gower distance that averages the distances per feature between samples. For numeric features it calculates the absolute distance divided by the range of the feature and for categorical features it is an indicator for whether the values are the same (see link for further details).

Imports

import pandas as pd
from sklearn.datasets import load_iris

from deepchecks.tabular import Dataset
from deepchecks.tabular.checks import OutlierSampleDetection

Prepare data

iris = pd.DataFrame(load_iris().data)
iris.describe()

	0	1	2	3
count	150.000000	150.000000	150.000000	150.000000
mean	5.843333	3.057333	3.758000	1.199333
std	0.828066	0.435866	1.765298	0.762238
min	4.300000	2.000000	1.000000	0.100000
25%	5.100000	2.800000	1.600000	0.300000
50%	5.800000	3.000000	4.350000	1.300000
75%	6.400000	3.300000	5.100000	1.800000
max	7.900000	4.400000	6.900000	2.500000

Add an outlier:

outlier_sample = [1, 10, 50, 100]
iris.loc[len(iris.index)] = outlier_sample
print(iris.tail())
modified_iris = Dataset(iris, cat_features=[])

       0     1     2      3
146  6.3   2.5   5.0    1.9
147  6.5   3.0   5.2    2.0
148  6.2   3.4   5.4    2.3
149  5.9   3.0   5.1    1.8
150  1.0  10.0  50.0  100.0

Run the Check

We define the nearest_neighbors_percent and the extent parameters for the LoOP algorithm.

check = OutlierSampleDetection(nearest_neighbors_percent=0.01, extent_parameter=3)
check.run(modified_iris)

Define a condition

Now, we define a condition that enforces that the ratio of outlier samples in out dataset is below 0.001.

check = OutlierSampleDetection()
check.add_condition_outlier_ratio_less_or_equal(max_outliers_ratio=0.001, outlier_score_threshold=0.9)
check.run(modified_iris)

previous

Class Imbalance

next

Identifier Label Correlation