Note
Click here to download the full example code
Model Error Analysis check#
This notebooks provides an overview for using and understanding the model error analysis check.
Structure:
What is the purpose of the check?#
Imports#
from deepchecks.vision.checks import ModelErrorAnalysis
Classification Performance Report#
Generate data and model:#
from deepchecks.vision.datasets.classification import mnist
mnist_model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')
Run the check:#
check = ModelErrorAnalysis(min_error_model_score=-0.1)
result = check.run(train_ds, test_ds, mnist_model)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Ingesting Batches - Train Dataset:
| | 0/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|########### | 11/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|###################### | 22/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|################################# | 33/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################################ | 44/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|####################################################### | 55/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|################################################################## | 66/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################################################################# | 77/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|######################################################################################## | 88/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|################################################################################################### | 99/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################################################################################################## | 110/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|######################################################################################################################### | 121/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|#################################################################################################################################### | 132/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################################################################################### | 143/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|########################################################################################################################################################## | 154/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|#############################################################################################################################################################| 157/157 [Time: 00:01]
Ingesting Batches - Test Dataset:
| | 0/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|# | 1/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|## | 2/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|### | 3/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|#### | 4/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##### | 5/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|###### | 6/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|####### | 7/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|######## | 8/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|######### | 9/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]
Computing Check:
| | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:122: FutureWarning:
Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:127: FutureWarning:
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Computing Check:
|#####| 1/1 [Time: 00:04]
Computing Check:
|#####| 1/1 [Time: 00:04]
If you have a GPU, you can speed up this check by passing it as an argument to .run() as device=<your GPU>
To display the results in an IDE like PyCharm, you can use the following code:
# result.show_in_window()
The result will be displayed in a new window.
Object Detection Class Performance#
For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds.
import numpy as np
from deepchecks.vision.datasets.detection import coco
Generate Data and Model#
We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model
yolo = coco.load_model(pretrained=True)
train_ds = coco.load_dataset(train=True, object_type='VisionData')
test_ds = coco.load_dataset(train=False, object_type='VisionData')
Run the check:#
check = ModelErrorAnalysis(min_error_model_score=-1)
result = check.run(train_ds, test_ds, yolo)
result
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:09]
Validating Input:
|#####| 1/1 [Time: 00:09]
Ingesting Batches - Train Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Train Dataset:
|##5 | 1/2 [Time: 00:06]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:12]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:12]
Ingesting Batches - Test Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##5 | 1/2 [Time: 00:05]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:12]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:12]
Computing Check:
| | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:122: FutureWarning:
Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:127: FutureWarning:
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Computing Check:
|#####| 1/1 [Time: 00:02]
Computing Check:
|#####| 1/1 [Time: 00:02]
Total running time of the script: ( 0 minutes 44.364 seconds)