.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/train_test_validation/plot_image_dataset_drift.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_train_test_validation_plot_image_dataset_drift.py: .. _plot_vision_image_dataset_drift: Image Dataset Drift ******************* This notebooks provides an overview for using and understanding the image dataset drift check, used to detect drift in simple image properties between train and test datasets. **Structure:** * `What Is Image Dataset Drift? <#what-is-image-dataset-drift>`__ * `Which Image Properties Are Used? <#which-image-properties-are-used>`__ * `Loading The Data <#loading-the-data>`__ * `Run The Check <#run-the-check>`__ * `Define a Condition <#define-a-condition>`__ What Is Image Dataset Drift? ---------------------------- Drift is simply a change in the distribution of data over time, and it is also one of the top reasons why machine learning model's performance degrades over time. Image dataset drift is a drift that occurs in more than one image property at a time, and may even affect the relationships between those properties, which are undetectable by univariate drift methods. For more information on drift, please visit our :doc:`drift guide `. How Deepchecks Detects Dataset Drift ------------------------------------ This check detects multivariate drift by using :ref:`a domain classifier `. Other methods to detect drift include :ref:`univariate measures ` which is used in other checks, such as :doc:`Image Property Drift check `. Using Properties to Detect Image Drift -------------------------------------- In computer vision specifically, we can't measure drift on the images directly, as the individual pixel has little value when estimating drift. Therefore, we calculate drift on different :doc:`properties of the image`, on which we can directly measure drift. Which Image Properties Are Used? -------------------------------- ============================== ========== Property name What is it ============================== ========== Aspect Ratio Ratio between height and width of image (height / width) Area Area of image in pixels (height * width) Brightness Average intensity of image pixels. Color channels have different weights according to RGB-to-Grayscale formula RMS Contrast Contrast of image, calculated by standard deviation of pixels Mean Red Relative Intensity Mean over all pixels of the red channel, scaled to their relative intensity in comparison to the other channels [r / (r + g + b)]. Mean Green Relative Intensity Mean over all pixels of the green channel, scaled to their relative intensity in comparison to the other channels [g / (r + g + b)]. Mean Blue Relative Intensity Mean over all pixels of the blue channel, scaled to their relative intensity in comparison to the other channels [b / (r + g + b)]. ============================== ========== .. GENERATED FROM PYTHON SOURCE LINES 67-69 Imports ------- .. GENERATED FROM PYTHON SOURCE LINES 69-75 .. code-block:: default import numpy as np from deepchecks.vision.checks import ImageDatasetDrift from deepchecks.vision.datasets.detection.coco import load_dataset .. GENERATED FROM PYTHON SOURCE LINES 76-78 Loading the data ---------------- .. GENERATED FROM PYTHON SOURCE LINES 78-83 .. code-block:: default train_ds = load_dataset(train=True, object_type='VisionData') test_ds = load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 84-88 Run the check ------------- without drift ^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 88-93 .. code-block:: default check = ImageDatasetDrift() result = check.run(train_dataset=train_ds, test_dataset=test_ds) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Dataset Drift


.. GENERATED FROM PYTHON SOURCE LINES 94-95 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 95-97 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 98-99 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 101-104 Insert drift ^^^^^^^^^^^^ Now, we will define a custom data object that will insert a drift to the training set. .. GENERATED FROM PYTHON SOURCE LINES 104-119 .. code-block:: default from deepchecks.vision.datasets.detection.coco import COCOData def add_brightness(img): reverse = 255 - img addition_of_brightness = (reverse * 0.07).astype(int) return img + addition_of_brightness class DriftedCOCO(COCOData): def batch_to_images(self, batch): return [add_brightness(np.array(img)) for img in batch[0]] .. GENERATED FROM PYTHON SOURCE LINES 120-126 .. code-block:: default train_dataloader = load_dataset(train=True, object_type='DataLoader') test_dataloader = load_dataset(train=False, object_type='DataLoader') drifted_train_ds = DriftedCOCO(train_dataloader) test_ds_coco = COCOData(test_dataloader) .. GENERATED FROM PYTHON SOURCE LINES 127-129 Run the check again ^^^^^^^^^^^^^^^^^^^ .. GENERATED FROM PYTHON SOURCE LINES 129-133 .. code-block:: default check = ImageDatasetDrift() result = check.run(train_dataset=drifted_train_ds, test_dataset=test_ds_coco) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:03] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:03] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Dataset Drift


.. GENERATED FROM PYTHON SOURCE LINES 134-139 Define a Condition ------------------ Now, we will define a condition that the maximum drift score is less than a certain threshold. In this example we will set the threshold at 0.2. In order to demonstrate the condition, we will use again the original (not drifted) train dataset. .. GENERATED FROM PYTHON SOURCE LINES 139-143 .. code-block:: default check = ImageDatasetDrift().add_condition_drift_score_less_than(0.2) result = check.run(train_dataset=train_ds, test_dataset=test_ds).show(show_additional_outputs=False) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Dataset Drift


.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 18.500 seconds) .. _sphx_glr_download_checks_gallery_vision_train_test_validation_plot_image_dataset_drift.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_image_dataset_drift.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_image_dataset_drift.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_