.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "checks_gallery/tabular/integrity/plot_outlier_sample_detection.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_checks_gallery_tabular_integrity_plot_outlier_sample_detection.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_checks_gallery_tabular_integrity_plot_outlier_sample_detection.py:


Outlier Sample Detection
***************

This notebooks provides an overview for using and understanding the Outlier Sample Detection check.

**Structure:**

* `How deepchecks detects outliers <#How-deepchecks-detects-outliers>`__
* `Prepare data <#prepare-data>`__
* `Run the check <#run-the-check>`__
* `Define a condition <#define-a-condition>`__

How deepchecks detects outliers
========================
Outlier Sample Detection searches for outliers samples (jointly across all features) using the LoOP algorithm.
The LoOP algorithm is a robust method for detecting outliers in a dataset across multiple variables by comparing
the density in the area of a sample with the densities in the areas of its nearest neighbors
(see `link <https://www.dbs.ifi.lmu.de/Publikationen/Papers/LoOP1649.pdf>`_ for further details).

LoOP relies on a distance matrix. In our implementation we use the Gower distance that averages the distances
per feature between samples. For numeric features it calculates the absolute distance divided by the range of the
feature and for categorical features it is an indicator whether the values are the same
(see `link <https://www.jstor.org/stable/2528823>`_ for further details).

.. GENERATED FROM PYTHON SOURCE LINES 29-31

Imports
=======

.. GENERATED FROM PYTHON SOURCE LINES 31-39

.. code-block:: default


    import pandas as pd
    from sklearn.datasets import load_iris

    from deepchecks.tabular import Dataset
    from deepchecks.tabular.checks.integrity.outlier_sample_detection import \
        OutlierSampleDetection








.. GENERATED FROM PYTHON SOURCE LINES 40-42

Prepare data
=========

.. GENERATED FROM PYTHON SOURCE LINES 42-46

.. code-block:: default


    iris = pd.DataFrame(load_iris().data)
    iris.describe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>0</th>
          <th>1</th>
          <th>2</th>
          <th>3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>count</th>
          <td>150.000000</td>
          <td>150.000000</td>
          <td>150.000000</td>
          <td>150.000000</td>
        </tr>
        <tr>
          <th>mean</th>
          <td>5.843333</td>
          <td>3.057333</td>
          <td>3.758000</td>
          <td>1.199333</td>
        </tr>
        <tr>
          <th>std</th>
          <td>0.828066</td>
          <td>0.435866</td>
          <td>1.765298</td>
          <td>0.762238</td>
        </tr>
        <tr>
          <th>min</th>
          <td>4.300000</td>
          <td>2.000000</td>
          <td>1.000000</td>
          <td>0.100000</td>
        </tr>
        <tr>
          <th>25%</th>
          <td>5.100000</td>
          <td>2.800000</td>
          <td>1.600000</td>
          <td>0.300000</td>
        </tr>
        <tr>
          <th>50%</th>
          <td>5.800000</td>
          <td>3.000000</td>
          <td>4.350000</td>
          <td>1.300000</td>
        </tr>
        <tr>
          <th>75%</th>
          <td>6.400000</td>
          <td>3.300000</td>
          <td>5.100000</td>
          <td>1.800000</td>
        </tr>
        <tr>
          <th>max</th>
          <td>7.900000</td>
          <td>4.400000</td>
          <td>6.900000</td>
          <td>2.500000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 47-48

Add an outlier:

.. GENERATED FROM PYTHON SOURCE LINES 48-54

.. code-block:: default


    outlier_sample = [1, 10, 50, 100]
    iris.loc[len(iris.index)] = outlier_sample
    print(iris.tail())
    modified_iris = Dataset(iris, cat_features=[])





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

           0     1     2      3
    146  6.3   2.5   5.0    1.9
    147  6.5   3.0   5.2    2.0
    148  6.2   3.4   5.4    2.3
    149  5.9   3.0   5.1    1.8
    150  1.0  10.0  50.0  100.0




.. GENERATED FROM PYTHON SOURCE LINES 55-58

Run the Check
=============
We define the nearest_neighbors_percent and the extent parameters for the LoOP algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 58-62

.. code-block:: default


    check = OutlierSampleDetection(nearest_neighbors_percent=0.01, extent_parameter=3)
    check.run(modified_iris)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">

    <script
        src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js"
        integrity="sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg=="
        crossorigin="anonymous"
        referrerpolicy="no-referrer">
    </script>

    <script type="text/javascript">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>
    <script type="text/javascript">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: "STIX-Web"}});}</script>
    <script type="text/javascript">
        if (typeof require !== 'undefined') {
            require.undef("plotly");
            requirejs.config({
                paths: {'plotly': ['https://cdn.plot.ly/plotly-2.8.3.min']}
            });
            require(
                ['plotly'],
                function(Plotly) {
                    window._Plotly = Plotly;
                    window.Plotly = Plotly;
                    console.log('Loaded plotly successfully');
                },
                function() {console.log('Failed to load plotly')}
            );
        } else {
            console.log('requirejs is not present');
        }
    </script>
    <h4><b>Outlier Sample Detection</b></h4><p>Detects outliers in a dataset using the LoOP algorithm.</p><h5><b>Additional Outputs</b></h5><div><span>
                        The Outlier Probability Score is calculated by the LoOP algorithm which measures the local deviation
                        of density of a given sample with respect to its neighbors. These outlier scores are directly
                        interpretable as a probability of an object being an outlier (see
                        <a href="https://www.dbs.ifi.lmu.de/Publikationen/Papers/LoOP1649.pdf"
                        target="_blank" rel="noopener noreferrer">link</a> for more information).<br><br>
                        </span></div><style type="text/css">
    #T_bc1f6 table {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_bc1f6 thead {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_bc1f6 tbody {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_bc1f6 th {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_bc1f6 td {
      text-align: left;
      white-space: pre-wrap;
    }
    </style>
    <table id="T_bc1f6">
      <thead>
        <tr>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_bc1f6_level0_col0" class="col_heading level0 col0" >Outlier Probability Score</th>
          <th id="T_bc1f6_level0_col1" class="col_heading level0 col1" >0</th>
          <th id="T_bc1f6_level0_col2" class="col_heading level0 col2" >1</th>
          <th id="T_bc1f6_level0_col3" class="col_heading level0 col3" >2</th>
          <th id="T_bc1f6_level0_col4" class="col_heading level0 col4" >3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_bc1f6_level0_row0" class="row_heading level0 row0" >150</th>
          <td id="T_bc1f6_row0_col0" class="data row0 col0" >1.00</td>
          <td id="T_bc1f6_row0_col1" class="data row0 col1" >1.00</td>
          <td id="T_bc1f6_row0_col2" class="data row0 col2" >10.00</td>
          <td id="T_bc1f6_row0_col3" class="data row0 col3" >50.00</td>
          <td id="T_bc1f6_row0_col4" class="data row0 col4" >100.00</td>
        </tr>
        <tr>
          <th id="T_bc1f6_level0_row1" class="row_heading level0 row1" >41</th>
          <td id="T_bc1f6_row1_col0" class="data row1 col0" >0.67</td>
          <td id="T_bc1f6_row1_col1" class="data row1 col1" >4.50</td>
          <td id="T_bc1f6_row1_col2" class="data row1 col2" >2.30</td>
          <td id="T_bc1f6_row1_col3" class="data row1 col3" >1.30</td>
          <td id="T_bc1f6_row1_col4" class="data row1 col4" >0.30</td>
        </tr>
        <tr>
          <th id="T_bc1f6_level0_row2" class="row_heading level0 row2" >108</th>
          <td id="T_bc1f6_row2_col0" class="data row2 col0" >0.50</td>
          <td id="T_bc1f6_row2_col1" class="data row2 col1" >6.70</td>
          <td id="T_bc1f6_row2_col2" class="data row2 col2" >2.50</td>
          <td id="T_bc1f6_row2_col3" class="data row2 col3" >5.80</td>
          <td id="T_bc1f6_row2_col4" class="data row2 col4" >1.80</td>
        </tr>
        <tr>
          <th id="T_bc1f6_level0_row3" class="row_heading level0 row3" >109</th>
          <td id="T_bc1f6_row3_col0" class="data row3 col0" >0.50</td>
          <td id="T_bc1f6_row3_col1" class="data row3 col1" >7.20</td>
          <td id="T_bc1f6_row3_col2" class="data row3 col2" >3.60</td>
          <td id="T_bc1f6_row3_col3" class="data row3 col3" >6.10</td>
          <td id="T_bc1f6_row3_col4" class="data row3 col4" >2.50</td>
        </tr>
        <tr>
          <th id="T_bc1f6_level0_row4" class="row_heading level0 row4" >22</th>
          <td id="T_bc1f6_row4_col0" class="data row4 col0" >0.44</td>
          <td id="T_bc1f6_row4_col1" class="data row4 col1" >4.60</td>
          <td id="T_bc1f6_row4_col2" class="data row4 col2" >3.60</td>
          <td id="T_bc1f6_row4_col3" class="data row4 col3" >1.00</td>
          <td id="T_bc1f6_row4_col4" class="data row4 col4" >0.20</td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 63-66

Define a condition
==================
Now, we define a condition that enforces that the ratio of outlier samples in out dataset is below 0.001.

.. GENERATED FROM PYTHON SOURCE LINES 66-69

.. code-block:: default


    check = OutlierSampleDetection()
    check.add_condition_outlier_ratio_not_greater_than(max_outliers_ratio=0.001, outlier_score_threshold=0.9)
    check.run(modified_iris)




.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">

    <script
        src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js"
        integrity="sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg=="
        crossorigin="anonymous"
        referrerpolicy="no-referrer">
    </script>

    <script type="text/javascript">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>
    <script type="text/javascript">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: "STIX-Web"}});}</script>
    <script type="text/javascript">
        if (typeof require !== 'undefined') {
            require.undef("plotly");
            requirejs.config({
                paths: {'plotly': ['https://cdn.plot.ly/plotly-2.8.3.min']}
            });
            require(
                ['plotly'],
                function(Plotly) {
                    window._Plotly = Plotly;
                    window.Plotly = Plotly;
                    console.log('Loaded plotly successfully');
                },
                function() {console.log('Failed to load plotly')}
            );
        } else {
            console.log('requirejs is not present');
        }
    </script>
    <h4><b>Outlier Sample Detection</b></h4><p>Detects outliers in a dataset using the LoOP algorithm.</p><h5><b>Conditions Summary</b></h5><style type="text/css">
    #T_66fc3 table {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_66fc3 thead {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_66fc3 tbody {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_66fc3 th {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_66fc3 td {
      text-align: left;
      white-space: pre-wrap;
    }
    </style>
    <table id="T_66fc3">
      <thead>
        <tr>
          <th id="T_66fc3_level0_col0" class="col_heading level0 col0" >Status</th>
          <th id="T_66fc3_level0_col1" class="col_heading level0 col1" >Condition</th>
          <th id="T_66fc3_level0_col2" class="col_heading level0 col2" >More Info</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td id="T_66fc3_row0_col0" class="data row0 col0" ><div style="color: orange;text-align: center;font-weight:bold">!</div></td>
          <td id="T_66fc3_row0_col1" class="data row0 col1" >Not more than 0.1% of dataset over outlier score 0.9</td>
          <td id="T_66fc3_row0_col2" class="data row0 col2" >0.6% of dataset samples above outlier threshold</td>
        </tr>
      </tbody>
    </table>
    <h5><b>Additional Outputs</b></h5><div><span>
                        The Outlier Probability Score is calculated by the LoOP algorithm which measures the local deviation
                        of density of a given sample with respect to its neighbors. These outlier scores are directly
                        interpretable as a probability of an object being an outlier (see
                        <a href="https://www.dbs.ifi.lmu.de/Publikationen/Papers/LoOP1649.pdf"
                        target="_blank" rel="noopener noreferrer">link</a> for more information).<br><br>
                        </span></div><style type="text/css">
    #T_c16c4 table {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_c16c4 thead {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_c16c4 tbody {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_c16c4 th {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_c16c4 td {
      text-align: left;
      white-space: pre-wrap;
    }
    </style>
    <table id="T_c16c4">
      <thead>
        <tr>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_c16c4_level0_col0" class="col_heading level0 col0" >Outlier Probability Score</th>
          <th id="T_c16c4_level0_col1" class="col_heading level0 col1" >0</th>
          <th id="T_c16c4_level0_col2" class="col_heading level0 col2" >1</th>
          <th id="T_c16c4_level0_col3" class="col_heading level0 col3" >2</th>
          <th id="T_c16c4_level0_col4" class="col_heading level0 col4" >3</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_c16c4_level0_row0" class="row_heading level0 row0" >150</th>
          <td id="T_c16c4_row0_col0" class="data row0 col0" >1.00</td>
          <td id="T_c16c4_row0_col1" class="data row0 col1" >1.00</td>
          <td id="T_c16c4_row0_col2" class="data row0 col2" >10.00</td>
          <td id="T_c16c4_row0_col3" class="data row0 col3" >50.00</td>
          <td id="T_c16c4_row0_col4" class="data row0 col4" >100.00</td>
        </tr>
        <tr>
          <th id="T_c16c4_level0_row1" class="row_heading level0 row1" >41</th>
          <td id="T_c16c4_row1_col0" class="data row1 col0" >0.67</td>
          <td id="T_c16c4_row1_col1" class="data row1 col1" >4.50</td>
          <td id="T_c16c4_row1_col2" class="data row1 col2" >2.30</td>
          <td id="T_c16c4_row1_col3" class="data row1 col3" >1.30</td>
          <td id="T_c16c4_row1_col4" class="data row1 col4" >0.30</td>
        </tr>
        <tr>
          <th id="T_c16c4_level0_row2" class="row_heading level0 row2" >108</th>
          <td id="T_c16c4_row2_col0" class="data row2 col0" >0.50</td>
          <td id="T_c16c4_row2_col1" class="data row2 col1" >6.70</td>
          <td id="T_c16c4_row2_col2" class="data row2 col2" >2.50</td>
          <td id="T_c16c4_row2_col3" class="data row2 col3" >5.80</td>
          <td id="T_c16c4_row2_col4" class="data row2 col4" >1.80</td>
        </tr>
        <tr>
          <th id="T_c16c4_level0_row3" class="row_heading level0 row3" >109</th>
          <td id="T_c16c4_row3_col0" class="data row3 col0" >0.50</td>
          <td id="T_c16c4_row3_col1" class="data row3 col1" >7.20</td>
          <td id="T_c16c4_row3_col2" class="data row3 col2" >3.60</td>
          <td id="T_c16c4_row3_col3" class="data row3 col3" >6.10</td>
          <td id="T_c16c4_row3_col4" class="data row3 col4" >2.50</td>
        </tr>
        <tr>
          <th id="T_c16c4_level0_row4" class="row_heading level0 row4" >22</th>
          <td id="T_c16c4_row4_col0" class="data row4 col0" >0.44</td>
          <td id="T_c16c4_row4_col1" class="data row4 col1" >4.60</td>
          <td id="T_c16c4_row4_col2" class="data row4 col2" >3.60</td>
          <td id="T_c16c4_row4_col3" class="data row4 col3" >1.00</td>
          <td id="T_c16c4_row4_col4" class="data row4 col4" >0.20</td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  0.070 seconds)


.. _sphx_glr_download_checks_gallery_tabular_integrity_plot_outlier_sample_detection.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_outlier_sample_detection.py <plot_outlier_sample_detection.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_outlier_sample_detection.ipynb <plot_outlier_sample_detection.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_