.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "checks_gallery/tabular/integrity/plot_mixed_data_types.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_checks_gallery_tabular_integrity_plot_mixed_data_types.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_checks_gallery_tabular_integrity_plot_mixed_data_types.py:


Mixed Data Types
****************

This notebooks provides an overview for using and understanding the mixed data types check.

**Structure:**

* `What are Mixed Data Types? <#what-are-mixed-data-types>`__
* `Run the Check <#run-the-check>`__
* `Define a Condition <#define-a-condition>`__

What are Mixed Data Types?
==========================
Mixed data types is when a column contains both string values and numeric values (either as numeric type or
as string like "42.90"). This may indicate a problem in the data collection pipeline, or represent a
problem situation for the model's training.

This checks searches for columns with a mix of strings and numeric values and returns them and their respective ratios.

Run the Check
=============
We will run the check on the adult dataset which can be downloaded from the
`UCI machine learning repository <http://archive.ics.uci.edu/ml>`_ and is also available in
`deepchecks.tabular.datasets`, and introduce to it some data type mixing in order to show the check's result.

.. GENERATED FROM PYTHON SOURCE LINES 28-61

.. code-block:: default


    import pandas as pd
    import numpy as np
    from deepchecks.tabular.datasets.classification import adult

    # Prepare functions to insert mixed data types

    def insert_new_values_types(col: pd.Series, ratio_to_replace: float, values_list):
        col = col.to_numpy().astype(object)
        indices_to_replace = np.random.choice(range(len(col)), int(len(col) * ratio_to_replace), replace=False)
        new_values = np.random.choice(values_list, len(indices_to_replace))
        col[indices_to_replace] = new_values
        return col


    def insert_string_types(col: pd.Series, ratio_to_replace):
        return insert_new_values_types(col, ratio_to_replace, ['a', 'b', 'c'])


    def insert_numeric_string_types(col: pd.Series, ratio_to_replace):
        return insert_new_values_types(col, ratio_to_replace, ['1.0', '1', '10394.33'])


    def insert_number_types(col: pd.Series, ratio_to_replace):
        return insert_new_values_types(col, ratio_to_replace, [66, 99.9])


    # Load dataset and insert some data type mixing
    adult_df, _ = adult.load_data(as_train_test=True, data_format='Dataframe')
    adult_df['workclass'] = insert_numeric_string_types(adult_df['workclass'], ratio_to_replace=0.01)
    adult_df['education'] = insert_number_types(adult_df['education'], ratio_to_replace=0.1)
    adult_df['age'] = insert_string_types(adult_df['age'], ratio_to_replace=0.5)








.. GENERATED FROM PYTHON SOURCE LINES 62-71

.. code-block:: default


    from deepchecks.tabular import Dataset
    from deepchecks.tabular.checks import MixedDataTypes

    adult_dataset = Dataset(adult_df, cat_features=['workclass', 'education'])
    check = MixedDataTypes()
    result = check.run(adult_dataset)
    result






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">

    <script
        src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js"
        integrity="sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg=="
        crossorigin="anonymous"
        referrerpolicy="no-referrer">
    </script>

    <script type="text/javascript">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>
    <script type="text/javascript">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: "STIX-Web"}});}</script>
    <script type="text/javascript">
        if (typeof require !== 'undefined') {
            require.undef("plotly");
            requirejs.config({
                paths: {'plotly': ['https://cdn.plot.ly/plotly-2.8.3.min']}
            });
            require(
                ['plotly'],
                function(Plotly) {
                    window._Plotly = Plotly;
                    window.Plotly = Plotly;
                    console.log('Loaded plotly successfully');
                },
                function() {console.log('Failed to load plotly')}
            );
        } else {
            console.log('requirejs is not present');
        }
    </script>
    <h4><b>Mixed Data Types</b></h4><p>Detect columns which contain a mix of numerical and string values.</p><h5><b>Additional Outputs</b></h5><div>* showing only the top 10 columns, you can change it using n_top_columns param</div><style type="text/css">
    #T_ffaf0 table {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ffaf0 thead {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ffaf0 tbody {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ffaf0 th {
      text-align: left;
      white-space: pre-wrap;
    }
    #T_ffaf0 td {
      text-align: left;
      white-space: pre-wrap;
    }
    </style>
    <table id="T_ffaf0">
      <thead>
        <tr>
          <th class="blank level0" >&nbsp;</th>
          <th id="T_ffaf0_level0_col0" class="col_heading level0 col0" >age</th>
          <th id="T_ffaf0_level0_col1" class="col_heading level0 col1" >workclass</th>
          <th id="T_ffaf0_level0_col2" class="col_heading level0 col2" >education</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th id="T_ffaf0_level0_row0" class="row_heading level0 row0" >strings</th>
          <td id="T_ffaf0_row0_col0" class="data row0 col0" >50%</td>
          <td id="T_ffaf0_row0_col1" class="data row0 col1" >99%</td>
          <td id="T_ffaf0_row0_col2" class="data row0 col2" >90%</td>
        </tr>
        <tr>
          <th id="T_ffaf0_level0_row1" class="row_heading level0 row1" >numbers</th>
          <td id="T_ffaf0_row1_col0" class="data row1 col0" >50%</td>
          <td id="T_ffaf0_row1_col1" class="data row1 col1" >1%</td>
          <td id="T_ffaf0_row1_col2" class="data row1 col2" >10%</td>
        </tr>
      </tbody>
    </table>

    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 72-79

Define a Condition
==================
We can define a condition that enforces the ratio of the "rare type" (the less common type, either numeric or string)
is not in a given range. The range represents the dangerous zone, when the ratio is lower than the lower bound, then
it's presumably a contamination but a negligible one, and when the ratio is higher than the upper bound, then it's
presumably supposed to contain both numbers and string values. So when the ratio is inside the range there is a real
chance that the rarer data type may represent a problem to model training and inference.

.. GENERATED FROM PYTHON SOURCE LINES 79-83

.. code-block:: default


    check = MixedDataTypes().add_condition_rare_type_ratio_not_in_range((0.01, 0.2))
    result = check.run(adult_dataset)
    result.show(show_additional_outputs=False)





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <!--
        ~ ----------------------------------------------------------------------------
        ~ Copyright (C) 2021-2022 Deepchecks (https://www.deepchecks.com)
        ~
        ~ This file is part of Deepchecks.
        ~ Deepchecks is distributed under the terms of the GNU Affero General
        ~ Public License (version 3 or later).
        ~ You should have received a copy of the GNU Affero General Public License
        ~ along with Deepchecks.  If not, see <http://www.gnu.org/licenses/>.
        ~ ----------------------------------------------------------------------------
        ~
    -->
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Mixed Data Types</title>
        <style>
            body {
                font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol';
                font-size: 1rem;
                line-height: 1.5;
                color: #212529;
                text-align: left;
                max-width: 1200px;
                margin: auto;
            }

            div.nbinput.container div.prompt *,
            div.nboutput.container div.prompt *,
            div.nbinput.container div.input_area pre,
            div.nboutput.container div.output_area pre,
            div.nbinput.container div.input_area .highlight,
            div.nboutput.container div.output_area .highlight {
                border: none;
                padding: 0;
                margin: 0;
                box-shadow: none;
            }

            div.nbinput.container > div[class*=highlight],
            div.nboutput.container > div[class*=highlight] {
                margin: 0;
            }

            div.nbinput.container div.prompt *,
            div.nboutput.container div.prompt * {
                background: none;
            }

            div.nboutput.container div.output_area .highlight,
            div.nboutput.container div.output_area pre {
                background: unset;
            }

            div.nboutput.container div.output_area div.highlight {
                color: unset;  /* override Pygments text color */
            }

            /* avoid gaps between output lines */
            div.nboutput.container div[class*=highlight] pre {
                line-height: normal;
            }

            /* input/output containers */
            div.nbinput.container,
            div.nboutput.container {
                display: -webkit-flex;
                display: flex;
                align-items: flex-start;
                margin: 0;
                width: 100%;
            }
            @media (max-width: 540px) {
                div.nbinput.container,
                div.nboutput.container {
                    flex-direction: column;
                }
            }

            /* input container */
            div.nbinput.container {
                padding-top: 5px;
            }

            /* last container */
            div.nblast.container {
                padding-bottom: 5px;
            }

            /* input prompt */
            div.nbinput.container div.prompt pre {
                color: #307FC1;
            }

            /* output prompt */
            div.nboutput.container div.prompt pre {
                color: #BF5B3D;
            }

            /* all prompts */
            div.nbinput.container div.prompt,
            div.nboutput.container div.prompt {
                width: 4.5ex;
                padding-top: 5px;
                position: relative;
                user-select: none;
            }

            div.nbinput.container div.prompt > div,
            div.nboutput.container div.prompt > div {
                position: absolute;
                right: 0;
                margin-right: 0.3ex;
            }

            @media (max-width: 540px) {
                div.nbinput.container div.prompt,
                div.nboutput.container div.prompt {
                    width: unset;
                    text-align: left;
                    padding: 0.4em;
                }
                div.nboutput.container div.prompt.empty {
                    padding: 0;
                }

                div.nbinput.container div.prompt > div,
                div.nboutput.container div.prompt > div {
                    position: unset;
                }
            }

            /* disable scrollbars on prompts */
            div.nbinput.container div.prompt pre,
            div.nboutput.container div.prompt pre {
                overflow: hidden;
            }

            /* input/output area */
            div.nbinput.container div.input_area,
            div.nboutput.container div.output_area {
                -webkit-flex: 1;
                flex: 1;
                overflow: auto;
            }
            @media (max-width: 540px) {
                div.nbinput.container div.input_area,
                div.nboutput.container div.output_area {
                    width: 100%;
                }
            }

            /* input area */
            div.nbinput.container div.input_area {
                border: 1px solid #e0e0e0;
                border-radius: 2px;
                /*background: #f5f5f5;*/
            }

            /* override MathJax center alignment in output cells */
            div.nboutput.container div[class*=MathJax] {
                text-align: left !important;
            }

            /* override sphinx.ext.imgmath center alignment in output cells */
            div.nboutput.container div.math p {
                text-align: left;
            }

            /* standard error */
            div.nboutput.container div.output_area.stderr {
                background: #fdd;
            }

            /* ANSI colors */
            .ansi-black-fg { color: #3E424D; }
            .ansi-black-bg { background-color: #3E424D; }
            .ansi-black-intense-fg { color: #282C36; }
            .ansi-black-intense-bg { background-color: #282C36; }
            .ansi-red-fg { color: #E75C58; }
            .ansi-red-bg { background-color: #E75C58; }
            .ansi-red-intense-fg { color: #B22B31; }
            .ansi-red-intense-bg { background-color: #B22B31; }
            .ansi-green-fg { color: #00A250; }
            .ansi-green-bg { background-color: #00A250; }
            .ansi-green-intense-fg { color: #007427; }
            .ansi-green-intense-bg { background-color: #007427; }
            .ansi-yellow-fg { color: #DDB62B; }
            .ansi-yellow-bg { background-color: #DDB62B; }
            .ansi-yellow-intense-fg { color: #B27D12; }
            .ansi-yellow-intense-bg { background-color: #B27D12; }
            .ansi-blue-fg { color: #208FFB; }
            .ansi-blue-bg { background-color: #208FFB; }
            .ansi-blue-intense-fg { color: #0065CA; }
            .ansi-blue-intense-bg { background-color: #0065CA; }
            .ansi-magenta-fg { color: #D160C4; }
            .ansi-magenta-bg { background-color: #D160C4; }
            .ansi-magenta-intense-fg { color: #A03196; }
            .ansi-magenta-intense-bg { background-color: #A03196; }
            .ansi-cyan-fg { color: #60C6C8; }
            .ansi-cyan-bg { background-color: #60C6C8; }
            .ansi-cyan-intense-fg { color: #258F8F; }
            .ansi-cyan-intense-bg { background-color: #258F8F; }
            .ansi-white-fg { color: #C5C1B4; }
            .ansi-white-bg { background-color: #C5C1B4; }
            .ansi-white-intense-fg { color: #A1A6B2; }
            .ansi-white-intense-bg { background-color: #A1A6B2; }

            .ansi-default-inverse-fg { color: #FFFFFF; }
            .ansi-default-inverse-bg { background-color: #000000; }

            .ansi-bold { font-weight: bold; }
            .ansi-underline { text-decoration: underline; }


            div.nbinput.container div.input_area div[class*=highlight] > pre,
            div.nboutput.container div.output_area div[class*=highlight] > pre,
            div.nboutput.container div.output_area div[class*=highlight].math,
            div.nboutput.container div.output_area.rendered_html,
            div.nboutput.container div.output_area > div.output_javascript,
            div.nboutput.container div.output_area:not(.rendered_html) > img{
                padding: 5px;
                margin: 0;
            }

            /* fix copybtn overflow problem in chromium (needed for 'sphinx_copybutton') */
            div.nbinput.container div.input_area > div[class^='highlight'],
            div.nboutput.container div.output_area > div[class^='highlight']{
                overflow-y: hidden;
            }

            /* hide copybtn icon on prompts (needed for 'sphinx_copybutton') */
            .prompt .copybtn {
                display: none;
            }

            /* Some additional styling taken form the Jupyter notebook CSS */
            div.rendered_html table {
            border: none;
            border-collapse: collapse;
            border-spacing: 0;
            color: black;
            font-size: 12px;
            table-layout: fixed;
            }
            div.rendered_html thead {
            border-bottom: 1px solid black;
            vertical-align: bottom;
            }
            div.rendered_html tr,
            div.rendered_html th,
            div.rendered_html td {
            text-align: right;
            vertical-align: middle;
            padding: 0.5em 0.5em;
            line-height: normal;
            white-space: normal;
            max-width: none;
            border: none;
            }
            div.rendered_html th {
            font-weight: bold;
            }
            div.rendered_html tbody tr:nth-child(odd) {
            background: #f5f5f5;
            }
            div.rendered_html tbody tr:hover {
            background: rgba(66, 165, 245, 0.2);
            }
        </style>
        <script>
            (!window.navigator.onLine ? alert('needs internet connection') : '');
        </script>
    </head>
    <body>
    

    <!-- Load require.js. Delete this if your page already loads require.js -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" crossorigin="anonymous"></script>
    <script src="https://unpkg.com/@jupyter-widgets/html-manager@^0.20.1/dist/embed-amd.js" crossorigin="anonymous"></script>

    <script type="application/vnd.jupyter.widget-state+json">
    {
      "version_major": 2,
      "version_minor": 0,
      "state": {
        "7651d7c4fa884d10840df23a28dab3a1": {
          "model_name": "VBoxModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [
              "rendered_html",
              "jp-RenderedHTMLCommon",
              "jp-RenderedHTML",
              "jp-OutputArea-output"
            ],
            "children": [
              "IPY_MODEL_c427ee6161dd47b3846abdc779be1eaf",
              "IPY_MODEL_0296277f3a35441c8dd35160fef6f1ce",
              "IPY_MODEL_40dbe1b772214cd4ae39c2f5dd9b723a"
            ],
            "layout": "IPY_MODEL_40c83433e3a14789a34fd99f013a4cdd"
          }
        },
        "c427ee6161dd47b3846abdc779be1eaf": {
          "model_name": "HTMLModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "layout": "IPY_MODEL_0d54a1e7c60a43e58b7cb48bbb11263a",
            "style": "IPY_MODEL_3df876228e2e4eb29d98309f51d6c6f7",
            "value": "<h4><b>Mixed Data Types</b></h4>"
          }
        },
        "0d54a1e7c60a43e58b7cb48bbb11263a": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        },
        "3df876228e2e4eb29d98309f51d6c6f7": {
          "model_name": "DescriptionStyleModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {}
        },
        "0296277f3a35441c8dd35160fef6f1ce": {
          "model_name": "HTMLModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "layout": "IPY_MODEL_0c6dd9237b6e4f4bb2cb8d4fed267f04",
            "style": "IPY_MODEL_8559852f4a794536aacdb693ca648150",
            "value": "<p>Detect columns which contain a mix of numerical and string values.</p>"
          }
        },
        "0c6dd9237b6e4f4bb2cb8d4fed267f04": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        },
        "8559852f4a794536aacdb693ca648150": {
          "model_name": "DescriptionStyleModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {}
        },
        "40dbe1b772214cd4ae39c2f5dd9b723a": {
          "model_name": "HTMLModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "layout": "IPY_MODEL_d64bb462bc744becb3d0fe8a7269e107",
            "style": "IPY_MODEL_e4e1befd4a494b1c8fd94c246bcc5e70",
            "value": "<h5><b>Conditions Summary</b></h5><style type=\"text/css\">\n#T_ad868 table {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_ad868 thead {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_ad868 tbody {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_ad868 th {\n  text-align: left;\n  white-space: pre-wrap;\n}\n#T_ad868 td {\n  text-align: left;\n  white-space: pre-wrap;\n}\n</style>\n<table id=\"T_ad868\">\n  <thead>\n    <tr>\n      <th id=\"T_ad868_level0_col0\" class=\"col_heading level0 col0\" >Status</th>\n      <th id=\"T_ad868_level0_col1\" class=\"col_heading level0 col1\" >Condition</th>\n      <th id=\"T_ad868_level0_col2\" class=\"col_heading level0 col2\" >More Info</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td id=\"T_ad868_row0_col0\" class=\"data row0 col0\" ><div style=\"color: orange;text-align: center;font-weight:bold\">!</div></td>\n      <td id=\"T_ad868_row0_col1\" class=\"data row0 col1\" >Rare data types in column are either more than 20% or less than 1% of the data</td>\n      <td id=\"T_ad868_row0_col2\" class=\"data row0 col2\" >Found columns with non-negligible quantities of samples with a different data type from the majority of samples: ['education']</td>\n    </tr>\n  </tbody>\n</table>\n"
          }
        },
        "d64bb462bc744becb3d0fe8a7269e107": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        },
        "e4e1befd4a494b1c8fd94c246bcc5e70": {
          "model_name": "DescriptionStyleModel",
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "state": {}
        },
        "40c83433e3a14789a34fd99f013a4cdd": {
          "model_name": "LayoutModel",
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "state": {}
        }
      }
    }
    </script>
    <script type="application/vnd.jupyter.widget-view+json">
    {"version_major": 2, "version_minor": 0, "model_id": "7651d7c4fa884d10840df23a28dab3a1"}
    </script>

    </body>
    </html>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  3.852 seconds)


.. _sphx_glr_download_checks_gallery_tabular_integrity_plot_mixed_data_types.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_mixed_data_types.py <plot_mixed_data_types.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_mixed_data_types.ipynb <plot_mixed_data_types.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_