Image Property Drift Check#

This notebooks provides an overview for using and understanding the image property drift check.

Structure:

How Does the ImagePropertyDrift Check Work?#

Data drift is simply a change in the distribution of data over time. It is also one of the top reasons that a machine learning model performance degrades over time.

In the context of machine learning, drift between the training set and the test set will likely make the model prone to errors. In other words, if the model was trained on data that is different from the current test data, it will probably make more mistakes predicting the target variable.

The Image Property Drift check calculates a drift score for each image property in the test dataset, by comparing its distribution to the train dataset. For this, we use the Earth Movers Distance (Wasserstein distance).

Which Image Properties Are Used?#

Property name

What is it

Aspect Ratio

Ratio between height and width of image (height / width)

Area

Area of image in pixels (height * width)

Brightness

Average intensity of image pixels. Color channels have different weights according to RGB-to-Grayscale formula

RMS Contrast

Contrast of image, calculated by standard deviation of pixels

Mean Red Relative Intensity

Mean over all pixels of the red channel, scaled to their relative intensity in comparison to the other channels [r / (r + g + b)].

Mean Green Relative Intensity

Mean over all pixels of the green channel, scaled to their relative intensity in comparison to the other channels [g / (r + g + b)].

Mean Blue Relative Intensity

Mean over all pixels of the blue channel, scaled to their relative intensity in comparison to the other channels [b / (r + g + b)].

Imports#

from deepchecks.vision.checks.distribution import ImagePropertyDrift
from deepchecks.vision.datasets.detection import coco

Prepare data#

from deepchecks.vision.utils import image_properties

train_dataset = coco.load_dataset(train=True, object_type='VisionData')
test_dataset = coco.load_dataset(train=False, object_type='VisionData')

Run the check#

check_result = ImagePropertyDrift().run(train_dataset, test_dataset)
check_result

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]


Ingesting Batches - Train Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset:  50%|# | 1/2 [00:01<00:01,  1.28s/ Batch]
Ingesting Batches - Train Dataset: 100%|##| 2/2 [00:02<00:00,  1.25s/ Batch]


Ingesting Batches - Test Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset:  50%|# | 1/2 [00:01<00:01,  1.09s/ Batch]
Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:02<00:00,  1.23s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00,  4.26 Check/s]

Image Property Drift

Calculate drift between train dataset and test dataset per image property, using statistical measures.

Additional Outputs
The Drift score is a measure for the difference between two distributions. In this check, drift is measured for the distribution of the following image properties: ['Area', 'Aspect Ratio', 'Brightness', 'Mean Blue Relative Intensity', 'Mean Green Relative Intensity', 'Mean Red Relative Intensity', 'RMS Contrast'].


Observe the check’s output#

The result value is a pandas DataFrame that contains drift score for each image property.

check_result.value

Out:

{'Aspect Ratio': 0.06673251751083462, 'Area': 0.05020210991879349, 'Brightness': 0.07114552630082199, 'RMS Contrast': 0.021987071714629717, 'Mean Red Relative Intensity': 0.03433695520859712, 'Mean Green Relative Intensity': 0.06265872287013945, 'Mean Blue Relative Intensity': 0.04916001505220028}

We can also pass the check a list of classes we wish to inspect, and the check will calculate the properties only for images either belonging to the classes or containing annotations belonging to the classes. (We’ll lower the min_samples to 5 to tell the check to calculate drift despite having only a few images left after the class filtration)

check_result = ImagePropertyDrift(classes_to_display=['bicycle', 'bench', 'bus', 'truck'], min_samples=5
                                  ).run(train_dataset, test_dataset)
check_result

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]


Ingesting Batches - Train Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset:  50%|# | 1/2 [00:00<00:00,  3.26 Batch/s]
Ingesting Batches - Train Dataset: 100%|##| 2/2 [00:00<00:00,  2.78 Batch/s]


Ingesting Batches - Test Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset:  50%|# | 1/2 [00:00<00:00,  2.94 Batch/s]
Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:00<00:00,  3.36 Batch/s]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00,  4.89 Check/s]

Image Property Drift

Calculate drift between train dataset and test dataset per image property, using statistical measures.

Additional Outputs
The Drift score is a measure for the difference between two distributions. In this check, drift is measured for the distribution of the following image properties: ['Area', 'Aspect Ratio', 'Brightness', 'Mean Blue Relative Intensity', 'Mean Green Relative Intensity', 'Mean Red Relative Intensity', 'RMS Contrast'].


Define a condition#

We can define a condition that make sure that image properties drift scores do not exceed allowed threshold.

check_result = (
    ImagePropertyDrift()
    .add_condition_drift_score_not_greater_than(0.001)
    .run(train_dataset, test_dataset)
)
check_result.show(show_additional_outputs=False)

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]


Ingesting Batches - Train Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset:  50%|# | 1/2 [00:01<00:01,  1.26s/ Batch]
Ingesting Batches - Train Dataset: 100%|##| 2/2 [00:02<00:00,  1.22s/ Batch]


Ingesting Batches - Test Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset:  50%|# | 1/2 [00:01<00:01,  1.12s/ Batch]
Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:02<00:00,  1.22s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00,  4.41 Check/s]
Image Property Drift


Check Parameters#

Image Property Drift Check accepts two parameters that allows us to control the look of the output:

  • image_properties - list of image properties that we are interested in

  • max_num_categories - Maximal number of categories to use for the calculation of drift using PSI (Population Stability Index)

Only next string values are allowed for the image_properties parameter:

  • aspect_ratio

  • area

  • brightness

  • mean_red_relative_intensity

  • mean_green_relative_intensity

  • mean_blue_relative_intensity

from typing import List

import numpy as np


def area(images: List[np.ndarray]) -> List[int]:
    # Return list of integers of image areas (height multiplied by width)
    return [img.shape[0] * img.shape[1] for img in images]


def aspect_ratio(images: List[np.ndarray]) -> List[float]:
    # Return list of floats of image height to width ratio
    return [img.shape[0] / img.shape[1] for img in images]


properties = [
    {'name': 'Area', 'method': area, 'output_type': 'continuous'},
    {'name': 'Aspect Ratio', 'method': aspect_ratio, 'output_type': 'continuous'}
]

check_result = ImagePropertyDrift(
    image_properties=properties,
    max_num_categories_for_drift=20
).run(train_dataset, test_dataset)

check_result

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]


Ingesting Batches - Train Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset:  50%|# | 1/2 [00:00<00:00,  4.70 Batch/s]
Ingesting Batches - Train Dataset: 100%|##| 2/2 [00:00<00:00,  4.89 Batch/s]


Ingesting Batches - Test Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset:  50%|# | 1/2 [00:00<00:00,  5.60 Batch/s]
Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:00<00:00,  5.25 Batch/s]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]

Image Property Drift

Calculate drift between train dataset and test dataset per image property, using statistical measures.

Additional Outputs
The Drift score is a measure for the difference between two distributions. In this check, drift is measured for the distribution of the following image properties: ['Area', 'Aspect Ratio'].


Total running time of the script: ( 0 minutes 13.137 seconds)

Gallery generated by Sphinx-Gallery