Weak Segments Performance#

This notebook provides an overview for using and understanding the weak segment performance check.

Structure:

What is the purpose of the check?#

The check is designed to easily identify the model’s weakest segments. The segments are characterized by the image properties such as contrast and aspect ratio.

Automatically detecting weak segments#

The check performs several steps:

  1. We calculate the image properties for each sample. The properties to calculate can be passed explicitly or resort to the default image properties.

  2. We calculate loss for each sample in the dataset using the provided model or predictions, the loss function can be passed explicitly or set to a default based on the task type.

  3. We train multiple simple tree based models, each one is trained using two properties to predict the per sample error calculated before.

  4. We extract the corresponding data samples for each of the leaves in each of the trees (data segments) and calculate the model performance on them. For the weakest data segments detected we also calculate the model’s performance on data segments surrounding them.

Generate Dataset#

Note

In this example, we use the pytorch version of the coco dataset and model. In order to run this example using tensorflow, please change the import statements to:

from deepchecks.vision.datasets.detection import coco_tensorflow as coco
from deepchecks.vision.checks import WeakSegmentsPerformance
from deepchecks.vision.datasets.detection import coco_torch as coco

coco_data = coco.load_dataset(train=False, object_type='VisionData')

Run the check#

check = WeakSegmentsPerformance()
result = check.run(coco_data)
result
Processing Batches:
|     | 0/1 [Time: 00:00]
Processing Batches:
|█████| 1/1 [Time: 00:00]
Processing Batches:
|█████| 1/1 [Time: 00:00]

Computing Check:
|     | 0/1 [Time: 00:00]

Computing Check:
|█████| 1/1 [Time: 00:00]

Computing Check:
|█████| 1/1 [Time: 00:00]
Weak Segments Performance


To display the results in an IDE like PyCharm, you can use the following code:

#  result.show_in_window()

The result will be displayed in a new window.

Observe the check’s output#

We see in the results that the check indeed found several segments on which the model performance is below average. In the heatmap display we can see the model’s performance on the weakest segments and their environment with respect to the two segmentation features. In order to get the full list of weak segments found we can look at the result.value attribute. Shown below are the 3 segments with the worst performance.

result.value['weak_segments_list'].head(3)
Mean IoU Feature1 Feature1 Range Feature2 Feature2 Range % of Data Samples in Segment
0 0.510713 Brightness (125.55977249145508, inf) RMS Contrast (-inf, 54.26441764831543) 7.81 [12, 24, 36, 42, 48]
1 0.516387 RMS Contrast (-inf, 69.78135681152344) Mean Red Relative Intensity (-inf, 0.3166673481464386) 10.94 [1, 3, 9, 10, 17, 36, 62]
2 0.518545 Aspect Ratio (0.66796875, inf) Mean Red Relative Intensity (0.32986657321453094, 0.34265194833278656) 12.50 [8, 31, 33, 37, 41, 45, 48, 60]


Now we will run a check with properties and minimum segment size ratio (the minimal fraction of the data to be considered as a segment) different from the defaults.

from deepchecks.vision.utils.image_properties import brightness, texture_level
properties = [{'name': 'brightness', 'method': brightness, 'output_type': 'numerical'},
              {'name': ' texture', 'method': texture_level, 'output_type': 'numerical'}]
check = WeakSegmentsPerformance(segment_minimum_size_ratio=0.03, image_properties=properties)
result = check.run(coco_data)
result.show()
Processing Batches:
|     | 0/1 [Time: 00:00]
Processing Batches:
|█████| 1/1 [Time: 00:00]
Processing Batches:
|█████| 1/1 [Time: 00:00]

Computing Check:
|     | 0/1 [Time: 00:00]

Computing Check:
|█████| 1/1 [Time: 00:00]

Computing Check:
|█████| 1/1 [Time: 00:00]
Weak Segments Performance


Define a condition#

We can add a condition that will validate the model’s performance on the weakest segment detected is above a certain threshold. A scenario where this can be useful is when we want to make sure that the model is not under performing on a subset of the data that is of interest to us.

# Let's add a condition and re-run the check:

check.add_condition_segments_relative_performance_greater_than(0.1)
result = check.run(coco_data)
result.show(show_additional_outputs=False)
Processing Batches:
|     | 0/1 [Time: 00:00]
Processing Batches:
|█████| 1/1 [Time: 00:00]
Processing Batches:
|█████| 1/1 [Time: 00:00]

Computing Check:
|     | 0/1 [Time: 00:00]

Computing Check:
|█████| 1/1 [Time: 00:00]

Computing Check:
|█████| 1/1 [Time: 00:00]
Weak Segments Performance


Total running time of the script: (0 minutes 4.449 seconds)

Gallery generated by Sphinx-Gallery