Class Performance#

This notebooks provides an overview for using and understanding the class performance check.

Structure:

What Is the Purpose of the Check?#

The class performance check evaluates several metrics on the given model and data and returns all of the results in a single check. The check uses the following default metrics:

Task Type

Property name

Classification

Precision

Classification

Recall

Object Detection

Average Precision

Object Detection

Average Recall

In addition to the default metrics, the check supports custom metrics that should be implemented using the torch.ignite.Metric API. These can be passed as a list using the alternative_metrics parameter of the check, which will override the default metrics.

Imports#

from deepchecks.vision.checks import ClassPerformance
from deepchecks.vision.datasets.classification import mnist

Classification Performance Report#

Generate data and model:#

mnist_model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')

Run the check#

check = ClassPerformance()
result = check.run(train_ds, test_ds, mnist_model)
result
Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]

Ingesting Batches - Train Dataset:
|                                                                                                                                                             | 0/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############                                                                                                                                              | 15/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################                                                                                                                              | 31/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################################                                                                                                              | 47/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################################################                                                                                              | 63/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################################################################                                                                              | 79/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################################################################################                                                              | 95/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|##############################################################################################################9                                              | 111/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################################################################################################################                              | 127/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###############################################################################################################################################              | 143/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|#############################################################################################################################################################| 157/157 [Time: 00:01]


Ingesting Batches - Test Dataset:
|          | 0/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##        | 2/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|####      | 4/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|######    | 6/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|########  | 8/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:00]



Computing Check:
|     | 0/1 [Time: 00:00]



Computing Check:
|#####| 1/1 [Time: 00:00]
Class Performance


If you have a GPU, you can speed up this check by calling:

check.run(train_ds, test_ds, mnist_model, device=<your GPU>)

To display the results in an IDE like PyCharm, you can use the following code:

result.show_in_window()
%%

The result will be displayed in a new window.

Object Detection Class Performance#

For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds.

from deepchecks.vision.datasets.detection import coco

Generate Data and Model#

We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model.

yolo = coco.load_model(pretrained=True)

train_ds = coco.load_dataset(train=True, object_type='VisionData')
test_ds = coco.load_dataset(train=False, object_type='VisionData')

Run the check#

check = ClassPerformance(show_only='best')
result = check.run(train_ds, test_ds, yolo)
result
Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:10]
Validating Input:
|#####| 1/1 [Time: 00:10]

Ingesting Batches - Train Dataset:
|     | 0/2 [Time: 00:00]

Ingesting Batches - Train Dataset:
|##5  | 1/2 [Time: 00:05]

Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:10]

Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:10]


Ingesting Batches - Test Dataset:
|     | 0/2 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##5  | 1/2 [Time: 00:05]


Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:10]


Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:10]



Computing Check:
|     | 0/1 [Time: 00:00]



Computing Check:
|#####| 1/1 [Time: 00:00]



Computing Check:
|#####| 1/1 [Time: 00:00]
Class Performance


If you have a GPU, you can speed up this check by calling:

# check.run(train_ds, test_ds, yolo, device=<your GPU>)

To display the results in an IDE like PyCharm, you can use the following code:

# result.show_in_window()

The result will be displayed in a new window.

Define a Condition#

We can also define a condition to validate that our model performance is above a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object.

check = ClassPerformance(show_only='worst')
check.add_condition_test_performance_greater_than(0.2)
result = check.run(train_ds, test_ds, yolo)
result
Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:10]
Validating Input:
|#####| 1/1 [Time: 00:10]

Ingesting Batches - Train Dataset:
|     | 0/2 [Time: 00:00]

Ingesting Batches - Train Dataset:
|##5  | 1/2 [Time: 00:05]

Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:10]

Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:10]


Ingesting Batches - Test Dataset:
|     | 0/2 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##5  | 1/2 [Time: 00:05]


Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:10]


Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:10]



Computing Check:
|     | 0/1 [Time: 00:00]



Computing Check:
|#####| 1/1 [Time: 00:00]



Computing Check:
|#####| 1/1 [Time: 00:00]
Class Performance


We detected that for several classes our model performance is below the threshold.

Total running time of the script: ( 1 minutes 5.144 seconds)

Gallery generated by Sphinx-Gallery