Note
Click here to download the full example code
Class Performance#
This notebooks provides an overview for using and understanding the class performance check.
Structure:
What Is the Purpose of the Check?#
The class performance check evaluates several metrics on the given model and data and returns all of the results in a single check. The check uses the following default metrics:
Task Type |
Property name |
|---|---|
Classification |
Precision |
Classification |
Recall |
Object Detection |
|
Object Detection |
In addition to the default metrics, the check supports custom metrics that should be implemented using the torch.ignite.Metric API. These can be passed as a list using the alternative_metrics parameter of the check, which will override the default metrics.
Imports#
from deepchecks.vision.checks import ClassPerformance
from deepchecks.vision.datasets.classification import mnist
Classification Performance Report#
Generate data and model:#
mnist_model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')
Run the check#
check = ClassPerformance()
result = check.run(train_ds, test_ds, mnist_model)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Ingesting Batches - Train Dataset:
| | 0/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|########## | 10/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|#################### | 20/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################## | 30/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|######################################## | 40/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|################################################## | 50/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################################################ | 60/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|###################################################################### | 70/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|################################################################################ | 80/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|########################################################################################## | 90/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|#################################################################################################### | 100/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################################################## | 110/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|######################################################################################################################### | 121/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|################################################################################################################################### | 131/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################################################################################# | 141/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|####################################################################################################################################################### | 151/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|#############################################################################################################################################################| 157/157 [Time: 00:01]
Ingesting Batches - Test Dataset:
| | 0/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|# | 1/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|## | 2/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|### | 3/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|#### | 4/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##### | 5/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|###### | 6/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|####### | 7/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|######## | 8/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|######### | 9/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]
Computing Check:
| | 0/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:00]
If you have a GPU, you can speed up this check by calling:
check.run(train_ds, test_ds, mnist_model, device=<your GPU>)
To display the results in an IDE like PyCharm, you can use the following code:
result.show_in_window()
- %%
The result will be displayed in a new window.
Object Detection Class Performance#
For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds.
from deepchecks.vision.datasets.detection import coco
Generate Data and Model#
We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model.
yolo = coco.load_model(pretrained=True)
train_ds = coco.load_dataset(train=True, object_type='VisionData')
test_ds = coco.load_dataset(train=False, object_type='VisionData')
Run the check#
check = ClassPerformance(show_only='best')
result = check.run(train_ds, test_ds, yolo)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:15]
Validating Input:
|#####| 1/1 [Time: 00:15]
Ingesting Batches - Train Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Train Dataset:
|##5 | 1/2 [Time: 00:07]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:15]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:15]
Ingesting Batches - Test Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##5 | 1/2 [Time: 00:07]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:15]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:15]
Computing Check:
| | 0/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:00]
If you have a GPU, you can speed up this check by calling:
# check.run(train_ds, test_ds, yolo, device=<your GPU>)
To display the results in an IDE like PyCharm, you can use the following code:
# result.show_in_window()
The result will be displayed in a new window.
Define a Condition#
We can also define a condition to validate that our model performance is above a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object.
check = ClassPerformance(show_only='worst')
check.add_condition_test_performance_greater_than(0.2)
result = check.run(train_ds, test_ds, yolo)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:15]
Validating Input:
|#####| 1/1 [Time: 00:15]
Ingesting Batches - Train Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Train Dataset:
|##5 | 1/2 [Time: 00:07]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:15]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:15]
Ingesting Batches - Test Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##5 | 1/2 [Time: 00:07]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:15]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:15]
Computing Check:
| | 0/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:00]
Computing Check:
|#####| 1/1 [Time: 00:00]
We detected that for several classes our model performance is below the threshold.
Total running time of the script: ( 1 minutes 38.623 seconds)