Note
Click here to download the full example code
Class Performance#
This notebooks provides an overview for using and understanding the class performance check.
Structure:
What Is the Purpose of the Check?#
The class performance check evaluates several metrics on the given model and data and returns all of the results in a single check. The check uses the following default metrics:
Task Type |
Property name |
|---|---|
Classification |
Precision |
Classification |
Recall |
Object Detection |
|
Object Detection |
In addition to the default metrics, the check supports custom metrics that should be implemented using the torch.ignite.Metric API. These can be passed as a list using the alternative_metrics parameter of the check, which will override the default metrics.
Imports#
from deepchecks.vision.checks.performance import ClassPerformance
from deepchecks.vision.datasets.classification import mnist
Classification Performance Report#
Generate data and model:#
mnist_model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')
Run the check#
check = ClassPerformance()
check.run(train_ds, test_ds, mnist_model)
Out:
Validating Input: 0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:00<00:00, 8.14 /s]
Ingesting Batches - Train Dataset: 0%| | 0/157 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset: 10%|################ | 16/157 [00:00<00:00, 150.82 Batch/s]
Ingesting Batches - Train Dataset: 20%|################################ | 32/157 [00:00<00:00, 152.44 Batch/s]
Ingesting Batches - Train Dataset: 31%|################################################ | 48/157 [00:00<00:00, 149.77 Batch/s]
Ingesting Batches - Train Dataset: 41%|################################################################ | 64/157 [00:00<00:00, 152.11 Batch/s]
Ingesting Batches - Train Dataset: 51%|################################################################################ | 80/157 [00:00<00:00, 153.46 Batch/s]
Ingesting Batches - Train Dataset: 61%|################################################################################################ | 96/157 [00:00<00:00, 154.04 Batch/s]
Ingesting Batches - Train Dataset: 71%|###############################################################################################################9 | 112/157 [00:00<00:00, 154.67 Batch/s]
Ingesting Batches - Train Dataset: 82%|################################################################################################################################ | 128/157 [00:00<00:00, 155.15 Batch/s]
Ingesting Batches - Train Dataset: 92%|################################################################################################################################################ | 144/157 [00:00<00:00, 155.58 Batch/s]
Ingesting Batches - Test Dataset: 0%| | 0/157 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset: 1%|## | 2/157 [00:00<00:14, 10.76 Batch/s]
Ingesting Batches - Test Dataset: 3%|#### | 4/157 [00:00<00:13, 10.96 Batch/s]
Ingesting Batches - Test Dataset: 4%|###### | 6/157 [00:00<00:13, 11.07 Batch/s]
Ingesting Batches - Test Dataset: 5%|######## | 8/157 [00:00<00:13, 11.09 Batch/s]
Ingesting Batches - Test Dataset: 6%|########## | 10/157 [00:00<00:13, 11.10 Batch/s]
Computing Check: 0%| | 0/1 [00:00<?, ? Check/s]
Object Detection Class Performance#
For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds.
from deepchecks.vision.datasets.detection import coco
Generate Data and Model#
We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model.
yolo = coco.load_model(pretrained=True)
train_ds = coco.load_dataset(train=True, object_type='VisionData')
test_ds = coco.load_dataset(train=False, object_type='VisionData')
Run the check#
check = ClassPerformance(show_only='best')
check.run(train_ds, test_ds, yolo)
Out:
Validating Input: 0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:11<00:00, 11.16s/ ]
Ingesting Batches - Train Dataset: 0%| | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset: 50%|# | 1/2 [00:05<00:05, 5.74s/ Batch]
Ingesting Batches - Train Dataset: 100%|##| 2/2 [00:11<00:00, 5.70s/ Batch]
Ingesting Batches - Test Dataset: 0%| | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset: 50%|# | 1/2 [00:05<00:05, 5.67s/ Batch]
Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:11<00:00, 5.67s/ Batch]
Computing Check: 0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00, 2.46 Check/s]
Define a Condition#
We can also define a condition to validate that our model performance is above a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object.
check = ClassPerformance(show_only='worst')
check.add_condition_test_performance_not_less_than(0.2)
result = check.run(train_ds, test_ds, yolo)
result
Out:
Validating Input: 0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:11<00:00, 11.13s/ ]
Ingesting Batches - Train Dataset: 0%| | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Train Dataset: 50%|# | 1/2 [00:05<00:05, 5.78s/ Batch]
Ingesting Batches - Train Dataset: 100%|##| 2/2 [00:11<00:00, 5.74s/ Batch]
Ingesting Batches - Test Dataset: 0%| | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches - Test Dataset: 50%|# | 1/2 [00:05<00:05, 5.72s/ Batch]
Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:11<00:00, 5.72s/ Batch]
Computing Check: 0%| | 0/1 [00:00<?, ? Check/s]
Computing Check: 100%|#| 1/1 [00:00<00:00, 2.46 Check/s]
We detected that for several classes our model performance is below the threshold.
Total running time of the script: ( 1 minutes 11.675 seconds)