• API Reference - WeakSegmentsPerformance

Weak Segments Performance

This notebook provides an overview for using and understanding the weak segment performance check.

Structure:

• What is the purpose of the check?

• Automatically detecting weak segments

• Generate data & model

• Run the check

• Define a condition

What is the purpose of the check?

The check is designed to easily identify the model’s weakest segments. The segments are characterized by the image properties such as contrast and aspect ratio.

Automatically detecting weak segments

The check performs several steps:

1. We calculate the image properties for each sample. The properties to calculate can be passed explicitly or resort to the default image properties.

2. We calculate loss for each sample in the dataset using the provided model or predictions, the loss function can be passed explicitly or set to a default based on the task type.

3. We train multiple simple tree based models, each one is trained using two properties to predict the per sample error calculated before.

4. We convert each of the leafs in each of the trees into a segment and calculate the segment’s performance. For the weakest segments detected we also calculate the model’s performance on the data segments surrounding them.

Run the check

from deepchecks.vision.checks import WeakSegmentsPerformance
from deepchecks.vision.datasets.detection import coco

coco_data = coco.load_dataset(train=False, object_type='VisionData')
model = coco.load_model()
check = WeakSegmentsPerformance()
result = check.run(coco_data, model)
result

Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:07]
Validating Input:
|#####| 1/1 [Time: 00:07]

Ingesting Batches:
|     | 0/2 [Time: 00:00]

Ingesting Batches:
|##5  | 1/2 [Time: 00:08]

Ingesting Batches:
|#####| 2/2 [Time: 00:18]

Ingesting Batches:
|#####| 2/2 [Time: 00:18]


Computing Check:
|     | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py:700: UserWarning:

A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.




Computing Check:
|#####| 1/1 [Time: 00:10]


Computing Check:
|#####| 1/1 [Time: 00:10]

If you have a GPU, you can speed up this check by passing it as an argument to .run() as device=<your GPU>

To display the results in an IDE like PyCharm, you can use the following code:

#  result.show_in_window()

The result will be displayed in a new window. %%

We see in the results that the check indeed found several segments on which the model performance is below average. In the heatmap display we can see the model’s performance on the weakest segments and their environment with respect to the two segmentation features. In order to get the full list of weak segments found we can look at the result.value attribute.

result.value['weak_segments_list']

	mean IoU score	Feature1	Feature1 range	Feature2	Feature2 range	% of data
2	0.567176	Aspect Ratio	(0.6875679194927216, 1.2355396747589111)	RMS Contrast	(0.2228790521621704, 0.2750105708837509)	15.6250
1	0.579779	Aspect Ratio	(0.6875679194927216, 1.2355396747589111)	Brightness	(0.45119504630565643, inf)	20.3125
3	0.582034	Aspect Ratio	(0.753125011920929, inf)	Mean Red Relative Intensity	(-inf, 0.3401498943567276)	15.6250
5	0.588011	Area	(273600.0, inf)	RMS Contrast	(0.24485763162374496, 0.27773164212703705)	17.1875
6	0.600899	Mean Red Relative Intensity	(0.3300568163394928, 0.3401498943567276)		None	15.6250
0	0.659983	Aspect Ratio	(0.6875679194927216, 1.2355396747589111)		None	48.4375
4	0.672503	Area	(279360.0, inf)	Brightness	(0.36843201518058777, 0.49187861382961273)	21.8750
7	0.674128	Brightness	(0.49187861382961273, inf)		None	26.5625

Now we will run a check with properties and minimum segment size ratio (the minimal fraction of the data to be considered as a segment) different from the defaults.

from deepchecks.vision.utils.image_properties import brightness, texture_level
properties = [{'name': 'brightness', 'method': brightness, 'output_type': 'numerical'},
              {'name': ' texture', 'method': texture_level, 'output_type': 'numerical'}]
check = WeakSegmentsPerformance(segment_minimum_size_ratio=0.03, image_properties=properties)
result = check.run(coco_data, model)
result.show_in_window()

Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:08]
Validating Input:
|#####| 1/1 [Time: 00:08]

Ingesting Batches:
|     | 0/2 [Time: 00:00]

Ingesting Batches:
|##5  | 1/2 [Time: 00:08]

Ingesting Batches:
|#####| 2/2 [Time: 00:16]

Ingesting Batches:
|#####| 2/2 [Time: 00:16]


Computing Check:
|     | 0/1 [Time: 00:00]


Computing Check:
|#####| 1/1 [Time: 00:00]


Computing Check:
|#####| 1/1 [Time: 00:00]

Define a condition

We can define on our check a condition that will validate that the ratio of the model performance on the weakest segment to the average model performance is less than a specified ratio.

# Let's add a condition and re-run the check:

check.add_condition_segments_relative_performance_greater_than(0.1)
result = check.run(coco_data, model)
result.show(show_additional_outputs=False)

Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:07]
Validating Input:
|#####| 1/1 [Time: 00:07]

Ingesting Batches:
|     | 0/2 [Time: 00:00]

Ingesting Batches:
|##5  | 1/2 [Time: 00:08]

Ingesting Batches:
|#####| 2/2 [Time: 00:16]

Ingesting Batches:
|#####| 2/2 [Time: 00:16]


Computing Check:
|     | 0/1 [Time: 00:00]


Computing Check:
|#####| 1/1 [Time: 00:00]


Computing Check:
|#####| 1/1 [Time: 00:00]

previous

Train Test Prediction Drift

next

Create a Custom Check