Mean Average Recall Report#

This notebooks provides an overview for using and understanding the mean average recall report check.

Structure:

What is the purpose of the check?#

The Mean Average Recall Report evaluates the mAR metric on the given model and data, and returns the mAR values per bounding box size category (small, medium, large). This check only works on the Object Detection task.

Imports#

import numpy as np

from deepchecks.vision.checks.performance import MeanAverageRecallReport
from deepchecks.vision.datasets.detection import coco

Generate Data and Model#

We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model.

For the label formatter - our dataset returns exactly the accepted format, so our formatting function is the simple lambda x: x function.

yolo = coco.load_model(pretrained=True)

test_ds = coco.load_dataset(train=False, object_type='VisionData')

Run the check#

check = MeanAverageRecallReport()
result = check.run(test_ds, yolo)
result

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:05<00:00,  5.22s/ ]


Ingesting Batches:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches:  50%|# | 1/2 [00:05<00:05,  5.77s/ Batch]
Ingesting Batches: 100%|##| 2/2 [00:11<00:00,  5.75s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00,  1.79 Check/s]

Mean Average Recall Report

Summarize mean average recall metrics on a dataset and model per detections and area range.

Additional Outputs
  AR@1 (%) AR@10 (%) AR@100 (%)
Area size      
All 0.33 0.42 0.43
Small (area < 32^2) 0.10 0.22 0.22
Medium (32^2 < area < 96^2) 0.33 0.42 0.42
Large (area < 96^2) 0.48 0.54 0.55


Observe the check’s output#

The result value is a dataframe that has the average recall score per each area range and IoU.

result.value
AR@1 (%) AR@10 (%) AR@100 (%)
Area size
All 0.330552 0.423444 0.429179
Small (area < 32^2) 0.104955 0.220594 0.220594
Medium (32^2 < area < 96^2) 0.325099 0.417392 0.423844
Large (area < 96^2) 0.481611 0.544408 0.549963


Define a condition#

We can define a condition that checks whether our model’s average recall score is not less than a given threshold

check = MeanAverageRecallReport().add_condition_test_average_recall_not_less_than(0.4)
result = check.run(test_ds, yolo)
result.show(show_additional_outputs=False)

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:05<00:00,  5.49s/ ]


Ingesting Batches:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches:  50%|# | 1/2 [00:05<00:05,  5.97s/ Batch]
Ingesting Batches: 100%|##| 2/2 [00:11<00:00,  5.85s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00,  1.83 Check/s]
Mean Average Recall Report


Total running time of the script: ( 0 minutes 35.458 seconds)

Gallery generated by Sphinx-Gallery