.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/train_test_validation/plot_image_property_drift.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_train_test_validation_plot_image_property_drift.py: .. _plot_vision_image_property_drift: Image Property Drift ******************** This notebooks provides an overview for using and understanding the image property drift check. **Structure:** * `What Is Image Drift? <#what-is-image-drift>`__ * `Which Image Properties Are Used? <#which-image-properties-are-used>`__ * `Prepare data <#prepare-data>`__ * `Run the check <#run-the-check>`__ * `Define a condition <#define-a-condition>`__ * `Check Parameters <#check-parameters>`__ What Is Image Drift? ================================= Drift is simply a change in the distribution of data over time, and it is also one of the top reasons why machine learning model's performance degrades over time. Image drift is a data drift that occurs in images in the dataset. For more information on drift, please visit our :doc:`drift guide `. How Deepchecks Detects Image Drift ------------------------------------ This check detects image property drift by using :ref:`univariate measures ` on each image property separately. Another possible method for drift detection is by :ref:`a domain classifier ` which is used in the :doc:`Image Dataset Drift check `. Using Properties to Detect Image Drift -------------------------------------------- In computer vision specifically, we can't measure drift on images directly, as the individual pixel has little value when estimating drift. Therefore, we calculate drift on different :doc:`properties of the image`, on which we can directly measure drift. Which Image Properties Are Used? ================================= ============================== ========== Property name What is it ============================== ========== Aspect Ratio Ratio between height and width of image (height / width) Area Area of image in pixels (height * width) Brightness Average intensity of image pixels. Color channels have different weights according to RGB-to-Grayscale formula RMS Contrast Contrast of image, calculated by standard deviation of pixels Mean Red Relative Intensity Mean over all pixels of the red channel, scaled to their relative intensity in comparison to the other channels [r / (r + g + b)]. Mean Green Relative Intensity Mean over all pixels of the green channel, scaled to their relative intensity in comparison to the other channels [g / (r + g + b)]. Mean Blue Relative Intensity Mean over all pixels of the blue channel, scaled to their relative intensity in comparison to the other channels [b / (r + g + b)]. ============================== ========== Imports ------- .. GENERATED FROM PYTHON SOURCE LINES 67-70 .. code-block:: default from deepchecks.vision.checks import ImagePropertyDrift from deepchecks.vision.datasets.detection import coco .. GENERATED FROM PYTHON SOURCE LINES 71-73 Prepare data ------------ .. GENERATED FROM PYTHON SOURCE LINES 73-78 .. code-block:: default from deepchecks.vision.utils import image_properties train_dataset = coco.load_dataset(train=True, object_type='VisionData') test_dataset = coco.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 79-81 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 81-85 .. code-block:: default check_result = ImagePropertyDrift().run(train_dataset, test_dataset) check_result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Property Drift


.. GENERATED FROM PYTHON SOURCE LINES 86-87 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 87-89 .. code-block:: default # check_result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 90-91 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 93-96 Observe the check’s output -------------------------- The result value is a pandas DataFrame that contains drift score for each image property. .. GENERATED FROM PYTHON SOURCE LINES 96-99 .. code-block:: default check_result.value .. rst-class:: sphx-glr-script-out .. code-block:: none {'Aspect Ratio': 0.06673251751083462, 'Area': 0.05020210991879349, 'Brightness': 0.07114552630082199, 'RMS Contrast': 0.021987071714629717, 'Mean Red Relative Intensity': 0.03433695520859712, 'Mean Green Relative Intensity': 0.06265872287013945, 'Mean Blue Relative Intensity': 0.04916001505220028} .. GENERATED FROM PYTHON SOURCE LINES 100-104 We can also pass the check a list of classes we wish to inspect, and the check will calculate the properties only for images either belonging to the classes or containing annotations belonging to the classes. (We'll lower the min_samples to 5 to tell the check to calculate drift despite having only a few images left after the class filtration) .. GENERATED FROM PYTHON SOURCE LINES 104-110 .. code-block:: default check_result = ImagePropertyDrift(classes_to_display=['person', 'traffic light'], min_samples=5 ).run(train_dataset, test_dataset) check_result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Property Drift


.. GENERATED FROM PYTHON SOURCE LINES 111-115 Define a condition ================== We can define a condition that make sure that image properties drift scores do not exceed allowed threshold. .. GENERATED FROM PYTHON SOURCE LINES 115-123 .. code-block:: default check_result = ( ImagePropertyDrift() .add_condition_drift_score_less_than(0.001) .run(train_dataset, test_dataset) ) check_result.show(show_additional_outputs=False) .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:01] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Property Drift


.. GENERATED FROM PYTHON SOURCE LINES 124-139 Check Parameters ---------------- Image Property Drift Check accepts two parameters that allows us to control the look of the output: * `vision_properties` - list of image properties that we are interested in * `max_num_categories` - Maximal number of categories to use for the calculation of drift using PSI (Population Stability Index) Only next string values are allowed for the `vision_properties` parameter: * `aspect_ratio` * `area` * `brightness` * `mean_red_relative_intensity` * `mean_green_relative_intensity` * `mean_blue_relative_intensity` .. GENERATED FROM PYTHON SOURCE LINES 139-165 .. code-block:: default from typing import List import numpy as np def area(images: List[np.ndarray]) -> List[int]: # Return list of integers of image areas (height multiplied by width) return [img.shape[0] * img.shape[1] for img in images] def aspect_ratio(images: List[np.ndarray]) -> List[float]: # Return list of floats of image height to width ratio return [img.shape[0] / img.shape[1] for img in images] properties = [ {'name': 'Area', 'method': area, 'output_type': 'numerical'}, {'name': 'Aspect Ratio', 'method': aspect_ratio, 'output_type': 'numerical'} ] check_result = ImagePropertyDrift( image_properties=properties, max_num_categories_for_drift=20 ).run(train_dataset, test_dataset) check_result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:00] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:00] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:00] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:00] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:00] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Image Property Drift


.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 18.588 seconds) .. _sphx_glr_download_checks_gallery_vision_train_test_validation_plot_image_property_drift.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_image_property_drift.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_image_property_drift.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_