.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "checks_gallery/tabular/integrity/plot_dominant_frequency_change.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_checks_gallery_tabular_integrity_plot_dominant_frequency_change.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_checks_gallery_tabular_integrity_plot_dominant_frequency_change.py:


Dominant Frequency Change
*************************
This example provides an overview for using and understanding the `Dominant Frequency Change` check.

**Structure:**

* `What is a Dominant Frequency Change? <#what-is-a-dominant-frequency-change>`__
* `Generate Data <#generate-data>`__
* `Run The Check <#run-the-check>`__
* `Define a Condition <#define-a-condition>`__

What is a Dominant Frequency Change?
====================================
Dominant Frequency Change is a data integrity check which simply checks whether dominant values have increased
significantly between test data and train data. Sharp changes in dominant values can indicate a problem with the
data collection or data processing pipeline (for example, a sharp incrase in a common null or constant value),
and will cause the model to fail to generalize well. This check goal is to catch these issues early in the pipeline.

This check compares the dominant values of each feature in the test data to the dominant values of the same feature in
the train data. If the ratio of the test to train dominant values is greater than a threshold, the check fails.
This threshold can be configured by specifying the `ratio_change_thres` parameter of the check.

The Definition of a Dominant Value
----------------------------------
The dominant value is defined as a value that is frequent in data at least more than ``dominance_ratio`` times from the
next most frequent value. The ``dominance_ratio`` is a configurable parameter of the check.

.. GENERATED FROM PYTHON SOURCE LINES 31-35

.. code-block:: default


    from deepchecks.tabular.checks.integrity import DominantFrequencyChange
    from deepchecks.tabular.datasets.classification import iris








.. GENERATED FROM PYTHON SOURCE LINES 36-38

Generate data
=============

.. GENERATED FROM PYTHON SOURCE LINES 38-40

.. code-block:: default

    train_ds, test_ds = iris.load_data(data_format='Dataset', as_train_test=True)








.. GENERATED FROM PYTHON SOURCE LINES 41-43

Introducing Duplicates in the Test Data
-----------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 43-48

.. code-block:: default


    # make duplicates in the test data
    test_ds.data.loc[test_ds.data.index % 2 == 0, 'petal length (cm)'] = 5.1
    test_ds.data.loc[test_ds.data.index / 3 > 8, 'sepal width (cm)'] = 2.7








.. GENERATED FROM PYTHON SOURCE LINES 49-51

Run The Check
=============

.. GENERATED FROM PYTHON SOURCE LINES 51-55

.. code-block:: default


    check = DominantFrequencyChange()
    check.run(test_ds, train_ds)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">

    <script
        src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js"
        integrity="sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg=="
        crossorigin="anonymous"
        referrerpolicy="no-referrer">
    </script>

    <script type="text/javascript">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>
    <script type="text/javascript">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: "STIX-Web"}});}</script>
    <script type="text/javascript">
        if (typeof require !== 'undefined') {
            require.undef("plotly");
            requirejs.config({
                paths: {'plotly': ['https://cdn.plot.ly/plotly-2.8.3.min']}
            });
            require(
                ['plotly'],
                function(Plotly) {
                    window._Plotly = Plotly;
                    window.Plotly = Plotly;
                    console.log('Loaded plotly successfully');
                },
                function() {console.log('Failed to load plotly')}
            );
        } else {
            console.log('requirejs is not present');
        }
    </script>
    <h4><b>Dominant Frequency Change</b></h4><p>Check if dominant values have increased significantly between test and reference data.</p><h5><b>Additional Outputs</b></h5><div>* showing only the top 10 columns, you can change it using n_top_columns param</div><style type="text/css">
    #T_ba7a5 table {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ba7a5 thead {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ba7a5 tbody {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ba7a5 th {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ba7a5 td {
      text-align: left;
      white-space: pre-wrap;
    }
    </style>
    <table id="T_ba7a5">
      <thead>
        <tr>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_ba7a5_level0_col0" class="col_heading level0 col0" >Value</th>
          <th id="T_ba7a5_level0_col1" class="col_heading level0 col1" >Train data %</th>
          <th id="T_ba7a5_level0_col2" class="col_heading level0 col2" >Test data %</th>
          <th id="T_ba7a5_level0_col3" class="col_heading level0 col3" >Train data #</th>
          <th id="T_ba7a5_level0_col4" class="col_heading level0 col4" >Test data #</th>
          <th id="T_ba7a5_level0_col5" class="col_heading level0 col5" >P value</th>
        </tr>
        <tr>
          <th class="index_name level0" >Column</th>
          <th class="blank col0" >&nbsp;</th>
          <th class="blank col1" >&nbsp;</th>
          <th class="blank col2" >&nbsp;</th>
          <th class="blank col3" >&nbsp;</th>
          <th class="blank col4" >&nbsp;</th>
          <th class="blank col5" >&nbsp;</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_ba7a5_level0_row0" class="row_heading level0 row0" >sepal width (cm)</th>
          <td id="T_ba7a5_row0_col0" class="data row0 col0" >2.70</td>
          <td id="T_ba7a5_row0_col1" class="data row0 col1" >0.37</td>
          <td id="T_ba7a5_row0_col2" class="data row0 col2" >0.06</td>
          <td id="T_ba7a5_row0_col3" class="data row0 col3" >14</td>
          <td id="T_ba7a5_row0_col4" class="data row0 col4" >7</td>
          <td id="T_ba7a5_row0_col5" class="data row0 col5" >0.00</td>
        </tr>
        <tr>
          <th id="T_ba7a5_level0_row1" class="row_heading level0 row1" >petal length (cm)</th>
          <td id="T_ba7a5_row1_col0" class="data row1 col0" >5.10</td>
          <td id="T_ba7a5_row1_col1" class="data row1 col1" >0.50</td>
          <td id="T_ba7a5_row1_col2" class="data row1 col2" >0.05</td>
          <td id="T_ba7a5_row1_col3" class="data row1 col3" >19</td>
          <td id="T_ba7a5_row1_col4" class="data row1 col4" >6</td>
          <td id="T_ba7a5_row1_col5" class="data row1 col5" >0.00</td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 56-58

Define a Condition
===================

.. GENERATED FROM PYTHON SOURCE LINES 58-62

.. code-block:: default

    check = DominantFrequencyChange()
    check.add_condition_ratio_of_change_not_greater_than(0.1)
    res = check.run(test_ds, train_ds)
    res.show(show_additional_outputs=False)





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <!--
        ~ ----------------------------------------------------------------------------
        ~ Copyright (C) 2021-2022 Deepchecks (https://www.deepchecks.com)
        ~
        ~ This file is part of Deepchecks.
        ~ Deepchecks is distributed under the terms of the GNU Affero General
        ~ Public License (version 3 or later).
        ~ You should have received a copy of the GNU Affero General Public License
        ~ along with Deepchecks.  If not, see <http://www.gnu.org/licenses/>.
        ~ ----------------------------------------------------------------------------
        ~
    -->
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Dominant Frequency Change</title>
        <style>
            body {
                font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';
                font-size: 1rem;
                line-height: 1.5;
                color: #212529;
                text-align: left;
                max-width: 1200px;
                margin: auto;
            }

            div.nbinput.container div.prompt *,
            div.nboutput.container div.prompt *,
            div.nbinput.container div.input_area pre,
            div.nboutput.container div.output_area pre,
            div.nbinput.container div.input_area .highlight,
            div.nboutput.container div.output_area .highlight {
                border: none;
                padding: 0;
                margin: 0;
                box-shadow: none;
            }

            div.nbinput.container > div[class*=highlight],
            div.nboutput.container > div[class*=highlight] {
                margin: 0;
            }

            div.nbinput.container div.prompt *,
            div.nboutput.container div.prompt * {
                background: none;
            }

            div.nboutput.container div.output_area .highlight,
            div.nboutput.container div.output_area pre {
                background: unset;
            }

            div.nboutput.container div.output_area div.highlight {
                color: unset;  /* override Pygments text color */
            }

            /* avoid gaps between output lines */
            div.nboutput.container div[class*=highlight] pre {
                line-height: normal;
            }

            /* input/output containers */
            div.nbinput.container,
            div.nboutput.container {
                display: -webkit-flex;
                display: flex;
                align-items: flex-start;
                margin: 0;
                width: 100%;
            }
            @media (max-width: 540px) {
                div.nbinput.container,
                div.nboutput.container {
                    flex-direction: column;
                }
            }

            /* input container */
            div.nbinput.container {
                padding-top: 5px;
            }

            /* last container */
            div.nblast.container {
                padding-bottom: 5px;
            }

            /* input prompt */
            div.nbinput.container div.prompt pre {
                color: #307FC1;
            }

            /* output prompt */
            div.nboutput.container div.prompt pre {
                color: #BF5B3D;
            }

            /* all prompts */
            div.nbinput.container div.prompt,
            div.nboutput.container div.prompt {
                width: 4.5ex;
                padding-top: 5px;
                position: relative;
                user-select: none;
            }

            div.nbinput.container div.prompt > div,
            div.nboutput.container div.prompt > div {
                position: absolute;
                right: 0;
                margin-right: 0.3ex;
            }

            @media (max-width: 540px) {
                div.nbinput.container div.prompt,
                div.nboutput.container div.prompt {
                    width: unset;
                    text-align: left;
                    padding: 0.4em;
                }
                div.nboutput.container div.prompt.empty {
                    padding: 0;
                }

                div.nbinput.container div.prompt > div,
                div.nboutput.container div.prompt > div {
                    position: unset;
                }
            }

            /* disable scrollbars on prompts */
            div.nbinput.container div.prompt pre,
            div.nboutput.container div.prompt pre {
                overflow: hidden;
            }

            /* input/output area */
            div.nbinput.container div.input_area,
            div.nboutput.container div.output_area {
                -webkit-flex: 1;
                flex: 1;
                overflow: auto;
            }
            @media (max-width: 540px) {
                div.nbinput.container div.input_area,
                div.nboutput.container div.output_area {
                    width: 100%;
                }
            }

            /* input area */
            div.nbinput.container div.input_area {
                border: 1px solid #e0e0e0;
                border-radius: 2px;
                /*background: #f5f5f5;*/
            }

            /* override MathJax center alignment in output cells */
            div.nboutput.container div[class*=MathJax] {
                text-align: left !important;
            }

            /* override sphinx.ext.imgmath center alignment in output cells */
            div.nboutput.container div.math p {
                text-align: left;
            }

            /* standard error */
            div.nboutput.container div.output_area.stderr {
                background: #fdd;
            }

            /* ANSI colors */
            .ansi-black-fg { color: #3E424D; }
            .ansi-black-bg { background-color: #3E424D; }
            .ansi-black-intense-fg { color: #282C36; }
            .ansi-black-intense-bg { background-color: #282C36; }
            .ansi-red-fg { color: #E75C58; }
            .ansi-red-bg { background-color: #E75C58; }
            .ansi-red-intense-fg { color: #B22B31; }
            .ansi-red-intense-bg { background-color: #B22B31; }
            .ansi-green-fg { color: #00A250; }
            .ansi-green-bg { background-color: #00A250; }
            .ansi-green-intense-fg { color: #007427; }
            .ansi-green-intense-bg { background-color: #007427; }
            .ansi-yellow-fg { color: #DDB62B; }
            .ansi-yellow-bg { background-color: #DDB62B; }
            .ansi-yellow-intense-fg { color: #B27D12; }
            .ansi-yellow-intense-bg { background-color: #B27D12; }
            .ansi-blue-fg { color: #208FFB; }
            .ansi-blue-bg { background-color: #208FFB; }
            .ansi-blue-intense-fg { color: #0065CA; }
            .ansi-blue-intense-bg { background-color: #0065CA; }
            .ansi-magenta-fg { color: #D160C4; }
            .ansi-magenta-bg { background-color: #D160C4; }
            .ansi-magenta-intense-fg { color: #A03196; }
            .ansi-magenta-intense-bg { background-color: #A03196; }
            .ansi-cyan-fg { color: #60C6C8; }
            .ansi-cyan-bg { background-color: #60C6C8; }
            .ansi-cyan-intense-fg { color: #258F8F; }
            .ansi-cyan-intense-bg { background-color: #258F8F; }
            .ansi-white-fg { color: #C5C1B4; }
            .ansi-white-bg { background-color: #C5C1B4; }
            .ansi-white-intense-fg { color: #A1A6B2; }
            .ansi-white-intense-bg { background-color: #A1A6B2; }

            .ansi-default-inverse-fg { color: #FFFFFF; }
            .ansi-default-inverse-bg { background-color: #000000; }

            .ansi-bold { font-weight: bold; }
            .ansi-underline { text-decoration: underline; }


            div.nbinput.container div.input_area div[class*=highlight] > pre,
            div.nboutput.container div.output_area div[class*=highlight] > pre,
            div.nboutput.container div.output_area div[class*=highlight].math,
            div.nboutput.container div.output_area.rendered_html,
            div.nboutput.container div.output_area > div.output_javascript,
            div.nboutput.container div.output_area:not(.rendered_html) > img{
                padding: 5px;
                margin: 0;
            }

            /* fix copybtn overflow problem in chromium (needed for 'sphinx_copybutton') */
            div.nbinput.container div.input_area > div[class^='highlight'],
            div.nboutput.container div.output_area > div[class^='highlight']{
                overflow-y: hidden;
            }

            /* hide copybtn icon on prompts (needed for 'sphinx_copybutton') */
            .prompt .copybtn {
                display: none;
            }

            /* Some additional styling taken form the Jupyter notebook CSS */
            div.rendered_html table {
            border: none;
            border-collapse: collapse;
            border-spacing: 0;
            color: black;
            font-size: 12px;
            table-layout: fixed;
            }
            div.rendered_html thead {
            border-bottom: 1px solid black;
            vertical-align: bottom;
            }
            div.rendered_html tr,
            div.rendered_html th,
            div.rendered_html td {
            text-align: right;
            vertical-align: middle;
            padding: 0.5em 0.5em;
            line-height: normal;
            white-space: normal;
            max-width: none;
            border: none;
            }
            div.rendered_html th {
            font-weight: bold;
            }
            div.rendered_html tbody tr:nth-child(odd) {
            background: #f5f5f5;
            }
            div.rendered_html tbody tr:hover {
            background: rgba(66, 165, 245, 0.2);
            }
        </style>
        <script>
            (!window.navigator.onLine ? alert('needs internet connection') : '');
        </script>
    </head>
    <body>
    

    <!-- Load require.js. Delete this if your page already loads require.js -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" crossorigin="anonymous"></script>
    <script src="https://unpkg.com/@jupyter-widgets/html-manager@^0.20.1/dist/embed-amd.js" crossorigin="anonymous"></script>

    <script type="application/vnd.jupyter.widget-state+json">
    {
      "version_major": 2,
      "version_minor": 0,
      "state": {
        "70955e5aac8b432985a06f0dccfe64f6": {
          "model_name": "VBoxModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [
              "rendered_html",
              "jp-RenderedHTMLCommon",
              "jp-RenderedHTML",
              "jp-OutputArea-output"
            ],
            "children": [
              "IPY_MODEL_d2840020bae042229e0c68328c279179",
              "IPY_MODEL_f646c445a85140c7b2195588052a7c3b",
              "IPY_MODEL_c9363a15b5fd410da9febe8d56119dc9"
            ],
            "layout": "IPY_MODEL_8818de83b1144869a8f6a9ddecca17c1"
          }
        },
        "d2840020bae042229e0c68328c279179": {
          "model_name": "HTMLModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "layout": "IPY_MODEL_3315742fb6564f7cac3dae58cbc831fb",
            "style": "IPY_MODEL_3e28cf77a9934da08a85c93254d3fb6d",
            "value": "<h4><b>Dominant Frequency Change</b></h4>"
          }
        },
        "3315742fb6564f7cac3dae58cbc831fb": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        },
        "3e28cf77a9934da08a85c93254d3fb6d": {
          "model_name": "DescriptionStyleModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {}
        },
        "f646c445a85140c7b2195588052a7c3b": {
          "model_name": "HTMLModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "layout": "IPY_MODEL_2ad9de6877134b43992f65c9f2b5884f",
            "style": "IPY_MODEL_e89223208af445e8a86105e273ddd3e0",
            "value": "<p>Check if dominant values have increased significantly between test and reference data.</p>"
          }
        },
        "2ad9de6877134b43992f65c9f2b5884f": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        },
        "e89223208af445e8a86105e273ddd3e0": {
          "model_name": "DescriptionStyleModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {}
        },
        "c9363a15b5fd410da9febe8d56119dc9": {
          "model_name": "HTMLModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "layout": "IPY_MODEL_c566d3131be641d38c0beb8d9da0127f",
            "style": "IPY_MODEL_a8926d10ed174a49915cd0499c880c3a",
            "value": "<h5><b>Conditions Summary</b></h5><style type=\"text/css\">\n#T_b9c80 table {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_b9c80 thead {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_b9c80 tbody {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_b9c80 th {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_b9c80 td {\n  text-align: left;\n  white-space: pre-wrap;\n}\n</style>\n<table id=\"T_b9c80\">\n  <thead>\n    <tr>\n      <th id=\"T_b9c80_level0_col0\" class=\"col_heading level0 col0\" >Status</th>\n      <th id=\"T_b9c80_level0_col1\" class=\"col_heading level0 col1\" >Condition</th>\n      <th id=\"T_b9c80_level0_col2\" class=\"col_heading level0 col2\" >More Info</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td id=\"T_b9c80_row0_col0\" class=\"data row0 col0\" ><div style=\"color: green;text-align: center\">\u2713</div></td>\n      <td id=\"T_b9c80_row0_col1\" class=\"data row0 col1\" >Change in ratio of dominant value in data is not greater than 10%</td>\n      <td id=\"T_b9c80_row0_col2\" class=\"data row0 col2\" ></td>\n    </tr>\n  </tbody>\n</table>\n"
          }
        },
        "c566d3131be641d38c0beb8d9da0127f": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        },
        "a8926d10ed174a49915cd0499c880c3a": {
          "model_name": "DescriptionStyleModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {}
        },
        "8818de83b1144869a8f6a9ddecca17c1": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        }
      }
    }
    </script>
    <script type="application/vnd.jupyter.widget-view+json">
    {"version_major": 2, "version_minor": 0, "model_id": "70955e5aac8b432985a06f0dccfe64f6"}
    </script>

    </body>
    </html>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  1.861 seconds)


.. _sphx_glr_download_checks_gallery_tabular_integrity_plot_dominant_frequency_change.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_dominant_frequency_change.py <plot_dominant_frequency_change.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_dominant_frequency_change.ipynb <plot_dominant_frequency_change.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_