Image Segment Performance#

This notebooks provides an overview for using and understanding image segment performance check.

Structure:

Why segment performance is important?#

The check helps to detect segments of your data that are under-performing based on the basic properties of the image. For example, by default the check would show how the performance depends on brightness, area and other such properties. Identifying your models’ weak segments might help to address specific issues and improve the overall performance of the model.

Run the check#

from deepchecks.vision.checks.performance import ImageSegmentPerformance
from deepchecks.vision.datasets.detection import coco

coco_data = coco.load_dataset(train=False, object_type='VisionData')
model = coco.load_model()

result = ImageSegmentPerformance().run(coco_data, model)
result

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:05<00:00,  5.29s/ ]


Ingesting Batches:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches:  50%|# | 1/2 [00:06<00:06,  6.12s/ Batch]
Ingesting Batches: 100%|##| 2/2 [00:12<00:00,  6.26s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:02<00:00,  2.94s/ Check]

Image Segment Performance

Segment the data by various properties of the image, and compare the performance of the segments.

Additional Outputs


Observe the check’s output#

The check segmented the data by different properties and calculated the metrics for each segment. As the value of result we return all the information on the different segments:

print(f'Properties: {result.value.keys()}')
print(f'brightness bins: {result.value["Brightness"]}')

Out:

Properties: dict_keys(['Aspect Ratio', 'Area', 'Brightness', 'RMS Contrast', 'Mean Red Relative Intensity', 'Mean Green Relative Intensity', 'Mean Blue Relative Intensity'])
brightness bins: [{'start': -inf, 'stop': 0.3283433811152742, 'count': 13, 'metrics': {'Average Precision': 0.4544554455445543, 'Average Recall': 0.4666666666666667}, 'display_range': '(-inf, 0.33)'}, {'start': 0.3283433811152742, 'stop': 0.4030219196204722, 'count': 13, 'metrics': {'Average Precision': 0.3029702970297029, 'Average Recall': 0.3}, 'display_range': '[0.33, 0.4)'}, {'start': 0.4030219196204722, 'stop': 0.4620466695414141, 'count': 12, 'metrics': {'Average Precision': 0.20495049504950488, 'Average Recall': 0.2}, 'display_range': '[0.4, 0.46)'}, {'start': 0.4620466695414141, 'stop': 0.5060984084251257, 'count': 13, 'metrics': {'Average Precision': 0.4544554455445543, 'Average Recall': 0.45}, 'display_range': '[0.46, 0.51)'}, {'start': 0.5060984084251257, 'stop': inf, 'count': 13, 'metrics': {'Average Precision': 0.2524752475247524, 'Average Recall': 0.25}, 'display_range': '[0.51, inf)'}]

Define a condition#

The check has a default condition which can be defined. The condition calculates for each property & metric the mean score and then looks at the ratio between the lowest segment score and the mean score. If this ratio is less than defined threshold, the condition fails.

The purpose of the condition is to catch properties segments that are significantly worse than the mean - which might indicate a problem.

check = ImageSegmentPerformance().add_condition_score_from_mean_ratio_not_less_than(0.5)
check.run(coco_data, model)

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:05<00:00,  5.22s/ ]


Ingesting Batches:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches:  50%|# | 1/2 [00:06<00:06,  6.13s/ Batch]
Ingesting Batches: 100%|##| 2/2 [00:12<00:00,  6.34s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:03<00:00,  3.19s/ Check]

Image Segment Performance

Segment the data by various properties of the image, and compare the performance of the segments.

Conditions Summary
Status Condition More Info
No segment with ratio between score to mean less than 50% Properties with failed segments: Area: {'Range': '(-inf, 243,100)', 'Metric': 'Average Precision', 'Ratio': 0.37}, Mean Blue Relative Intensity: {'Range': '[0.28, 0.32)', 'Metric': 'Average Precision', 'Ratio': 0.35}, Mean Red Relative Intensity: {'Range': '[0.33, 0.34)', 'Metric': 'Average Recall', 'Ratio': 0.19}
Additional Outputs


In this case the condition identified under-performing segments in the properties: mean_blue_relative_intensity, brightness, aspect_ratio, mean_red_relative_intensity

Total running time of the script: ( 0 minutes 42.137 seconds)

Gallery generated by Sphinx-Gallery