Note
Click here to download the full example code
Image Segment Performance#
This notebooks provides an overview for using and understanding image segment performance check.
Structure:
Why segment performance is important?#
The check helps to detect segments of your data that are under-performing based on the basic properties of the image. For example, by default the check would show how the performance depends on brightness, area and other such properties. Identifying your models’ weak segments might help to address specific issues and improve the overall performance of the model.
Run the check#
from deepchecks.vision.checks import ImageSegmentPerformance
from deepchecks.vision.datasets.detection import coco
coco_data = coco.load_dataset(train=False, object_type='VisionData')
model = coco.load_model()
result = ImageSegmentPerformance().run(coco_data, model)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:04]
Validating Input:
|#####| 1/1 [Time: 00:04]
Ingesting Batches:
| | 0/2 [Time: 00:00]
Ingesting Batches:
|##5 | 1/2 [Time: 00:05]
Ingesting Batches:
|#####| 2/2 [Time: 00:12]
Ingesting Batches:
|#####| 2/2 [Time: 00:12]
Computing Check:
| | 0/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:02]
Computing Check:
|#####| 1/1 [Time: 00:02]
If you have a GPU, you can speed up this check by passing it as an argument to .run() as device=<your GPU>
To display the results in an IDE like PyCharm, you can use the following code:
# result.show_in_window()
The result will be displayed in a new window.
Observe the check’s output#
The check segmented the data by different properties and calculated the metrics for each segment. As the value of result we return all the information on the different segments:
print(f'Properties: {result.value.keys()}')
print(f'brightness bins: {result.value["Brightness"]}')
Properties: dict_keys(['Aspect Ratio', 'Area', 'Brightness', 'RMS Contrast', 'Mean Red Relative Intensity', 'Mean Green Relative Intensity', 'Mean Blue Relative Intensity'])
brightness bins: [{'start': -inf, 'stop': 0.3283433811152742, 'count': 13, 'metrics': {'Average Precision': 0.4544554455445543, 'Average Recall': 0.4666666666666667}, 'display_range': '(-inf, 0.33)'}, {'start': 0.3283433811152742, 'stop': 0.4030219196204722, 'count': 13, 'metrics': {'Average Precision': 0.3029702970297029, 'Average Recall': 0.3}, 'display_range': '[0.33, 0.4)'}, {'start': 0.4030219196204722, 'stop': 0.4620466695414141, 'count': 12, 'metrics': {'Average Precision': 0.20495049504950488, 'Average Recall': 0.2}, 'display_range': '[0.4, 0.46)'}, {'start': 0.4620466695414141, 'stop': 0.5060984084251257, 'count': 13, 'metrics': {'Average Precision': 0.4544554455445543, 'Average Recall': 0.45}, 'display_range': '[0.46, 0.51)'}, {'start': 0.5060984084251257, 'stop': inf, 'count': 13, 'metrics': {'Average Precision': 0.2524752475247524, 'Average Recall': 0.25}, 'display_range': '[0.51, inf)'}]
Define a condition#
The check has a default condition which can be defined. The condition calculates for each property & metric the mean score and then looks at the ratio between the lowest segment score and the mean score. If this ratio is less than defined threshold, the condition fails.
The purpose of the condition is to catch properties segments that are significantly worse than the mean - which might indicate a problem.
check = ImageSegmentPerformance().add_condition_score_from_mean_ratio_greater_than(0.5)
result = check.run(coco_data, model)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:04]
Validating Input:
|#####| 1/1 [Time: 00:04]
Ingesting Batches:
| | 0/2 [Time: 00:00]
Ingesting Batches:
|##5 | 1/2 [Time: 00:05]
Ingesting Batches:
|#####| 2/2 [Time: 00:12]
Ingesting Batches:
|#####| 2/2 [Time: 00:12]
Computing Check:
| | 0/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:02]
Computing Check:
|#####| 1/1 [Time: 00:02]
In this case the condition identified under-performing segments in the properties: mean_blue_relative_intensity, brightness, aspect_ratio, mean_red_relative_intensity
Total running time of the script: ( 0 minutes 39.596 seconds)