.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "vision/auto_checks/train_test_validation/plot_label_drift.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_vision_auto_checks_train_test_validation_plot_label_drift.py: .. _vision__label_drift: Label Drift ********************** This notebooks provides an overview for using and understanding label drift check. **Structure:** * `What Is Label Drift? <#what-is-label-drift>`__ * `Which Label Properties Are Used? <#which-label-properties-are-used>`__ * `Run check on a Classification task <#run-the-check-on-a-classification-task-mnist>`__ * `Run check on an Object Detection task <#run-the-check-on-an-object-detection-task-coco>`__ What Is Label Drift? ======================== Drift is simply a change in the distribution of data over time, and it is also one of the top reasons why machine learning model's performance degrades over time. Label drift is when drift occurs in the label itself. For more information on drift, please visit our :ref:`drift_user_guide`. How Deepchecks Detects Label Drift ------------------------------------ This check detects label drift by using :ref:`univariate measures ` on the label properties. Using Label Properties to Detect Label Drift -------------------------------------------- In computer vision specifically, our labels may be complex, and measuring their drift is not a straightforward task. Therefore, we calculate drift on different :ref:`properties of the label`, on which we can directly measure drift. Which Label Properties Are Used? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ================ =================================== ========== Task Type Property name What is it ================ =================================== ========== Classification Samples Per Class Number of images per class Object Detection Samples Per Class Number of bounding boxes per class Object Detection Bounding Box Area Area of bounding box (height * width) Object Detection Number of Bounding Boxes Per Image Number of bounding box objects in each image ================ =================================== ========== Run the check on a Classification task (MNIST) ============================================== Imports ------- .. GENERATED FROM PYTHON SOURCE LINES 59-64 .. note:: In this example, we use the pytorch version of the mnist dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.classification.mnist_tensorflow import load_dataset .. GENERATED FROM PYTHON SOURCE LINES 65-69 .. code-block:: default from deepchecks.vision.checks import LabelDrift from deepchecks.vision.datasets.classification.mnist_torch import load_dataset .. GENERATED FROM PYTHON SOURCE LINES 70-72 Loading Data ------------ .. GENERATED FROM PYTHON SOURCE LINES 72-77 .. code-block:: default train_ds = load_dataset(train=True, batch_size=64, object_type='VisionData') test_ds = load_dataset(train=False, batch_size=1000, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 78-80 Running LabelDrift on classification --------------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 80-85 .. code-block:: default check = LabelDrift() result = check.run(train_ds, test_ds) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:04] Processing Test Batches: |█████| 1/1 [Time: 00:04] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Label Drift


.. GENERATED FROM PYTHON SOURCE LINES 86-91 Understanding the results ------------------------- We can see there is almost no drift between the train & test labels. This means the split to train and test was good (as it is balanced and random). Let's check the performance of a simple model trained on MNIST. .. GENERATED FROM PYTHON SOURCE LINES 91-96 .. code-block:: default from deepchecks.vision.checks import ClassPerformance ClassPerformance().run(train_ds, test_ds) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: |█████| 1/1 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 97-98 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 98-100 .. code-block:: default # ClassPerformance().run(train_ds, test_ds, mnist_model).show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 101-102 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 104-110 MNIST with label drift ====================== Now, let's try to separate the MNIST dataset in a different manner that will result in a label drift, and see how it affects the performance. We are going to create a custom `collate_fn`` in the test dataset, that will select samples with class 0 with a probability of 1/10. .. GENERATED FROM PYTHON SOURCE LINES 112-114 Inserting drift to the test set ------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 114-138 .. code-block:: default import numpy as np np.random.seed(42) def generate_collate_fn_with_label_drift(collate_fn): def collate_fn_with_label_drift(batch): batch_dict = collate_fn(batch) images = batch_dict['images'] labels = batch_dict['labels'] for i in range(len(images)): image, label = images[i], labels[i] if label == 0: if np.random.randint(5) != 0: batch_dict['labels'][i] = 1 return batch_dict return collate_fn_with_label_drift mod_test_ds = load_dataset(train=False, batch_size=1000, object_type='VisionData') mod_test_ds._batch_loader.collate_fn = generate_collate_fn_with_label_drift(mod_test_ds._batch_loader.collate_fn) .. GENERATED FROM PYTHON SOURCE LINES 139-141 Run the check ============= .. GENERATED FROM PYTHON SOURCE LINES 141-145 .. code-block:: default check = LabelDrift() check.run(train_ds, mod_test_ds) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:04] Processing Test Batches: |█████| 1/1 [Time: 00:04] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Label Drift


.. GENERATED FROM PYTHON SOURCE LINES 146-150 Add a condition --------------- We could also add a condition to the check to alert us to changes in the label distribution, such as the one that occurred here. .. GENERATED FROM PYTHON SOURCE LINES 150-156 .. code-block:: default check = LabelDrift().add_condition_drift_score_less_than() check.run(train_ds, mod_test_ds) # As we can see, the condition alerts us to the presence of drift in the label. .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: |█████| 1/1 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Label Drift


.. GENERATED FROM PYTHON SOURCE LINES 157-161 Results ------- We can see the check successfully detects the (expected) drift in class 0 distribution between the train and test sets .. GENERATED FROM PYTHON SOURCE LINES 163-165 But how does this affect the performance of the model? ------------------------------------------------------ .. GENERATED FROM PYTHON SOURCE LINES 165-168 .. code-block:: default ClassPerformance().run(train_ds, mod_test_ds) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: |█████| 1/1 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 169-172 Understanding the results ------------------------------- We can see the drop in the precision of class 0, which was caused by the class imbalance indicated earlier by the label drift check. .. GENERATED FROM PYTHON SOURCE LINES 174-182 Run the check on an Object Detection task (COCO) ================================================ .. note:: In this example, we use the pytorch version of the coco dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.detection.coco_tensorflow import load_dataset .. GENERATED FROM PYTHON SOURCE LINES 182-188 .. code-block:: default from deepchecks.vision.datasets.detection.coco_torch import load_dataset train_ds = load_dataset(train=True, object_type='VisionData') test_ds = load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 189-193 .. code-block:: default check = LabelDrift() check.run(train_ds, test_ds) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Label Drift


.. GENERATED FROM PYTHON SOURCE LINES 194-200 Label drift is detected! ------------------------ We can see that the COCO128 contains a drift in the out of the box dataset. In addition to the label count per class, the label drift check for object detection tasks include drift calculation on certain measurements, like the bounding box area and the number of bboxes per image. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 22.448 seconds) .. _sphx_glr_download_vision_auto_checks_train_test_validation_plot_label_drift.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_label_drift.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_label_drift.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_