Note
Click here to download the full example code
Model Error Analysis check#
The ModelErrorAnalysis check is deprecated, please use WeakSegmentsPerformance instead.
This notebook provides an overview for using and understanding the model error analysis check.
Structure:
What is the purpose of the check?#
Imports#
from deepchecks.vision.checks import ModelErrorAnalysis
Classification Performance Report#
Generate data and model:#
from deepchecks.vision.datasets.classification import mnist
mnist_model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')
Run the check:#
check = ModelErrorAnalysis(min_error_model_score=-0.1)
result = check.run(train_ds, test_ds, mnist_model)
result
/home/runner/work/deepchecks/deepchecks/deepchecks/vision/checks/model_evaluation/model_error_analysis.py:81: DeprecationWarning:
The ModelErrorAnalysis check is deprecated and will be removed in the 0.11 version. Please use the WeakSegmentsPerformance check instead.
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Ingesting Batches - Train Dataset:
| | 0/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|######9 | 7/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############### | 15/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|####################### | 23/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################### | 31/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|####################################### | 39/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################################### | 47/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|####################################################### | 55/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|############################################################### | 63/157 [Time: 00:00]
Ingesting Batches - Train Dataset:
|####################################################################### | 71/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################### | 79/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|####################################################################################### | 87/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################################### | 95/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|####################################################################################################### | 103/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|##############################################################################################################9 | 111/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|####################################################################################################################### | 119/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################################################################### | 127/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|####################################################################################################################################### | 135/157 [Time: 00:01]
Ingesting Batches - Train Dataset:
|############################################################################################################################################### | 143/157 [Time: 00:02]
Ingesting Batches - Train Dataset:
|####################################################################################################################################################### | 151/157 [Time: 00:02]
Ingesting Batches - Train Dataset:
|#############################################################################################################################################################| 157/157 [Time: 00:02]
Ingesting Batches - Test Dataset:
| | 0/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|# | 1/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|## | 2/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|### | 3/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|#### | 4/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##### | 5/10 [Time: 00:00]
Ingesting Batches - Test Dataset:
|###### | 6/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|####### | 7/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|######## | 8/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|######### | 9/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]
Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]
Computing Check:
| | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:92: FutureWarning:
Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:97: FutureWarning:
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Computing Check:
|#####| 1/1 [Time: 00:04]
Computing Check:
|#####| 1/1 [Time: 00:04]
If you have a GPU, you can speed up this check by passing it as an argument to .run() as device=<your GPU>
To display the results in an IDE like PyCharm, you can use the following code:
# result.show_in_window()
The result will be displayed in a new window.
Object Detection Class Performance#
For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds.
import numpy as np
from deepchecks.vision.datasets.detection import coco
Generate Data and Model#
We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model
yolo = coco.load_model(pretrained=True)
train_ds = coco.load_dataset(train=True, object_type='VisionData')
test_ds = coco.load_dataset(train=False, object_type='VisionData')
Run the check:#
check = ModelErrorAnalysis(min_error_model_score=-1)
result = check.run(train_ds, test_ds, yolo)
result
/home/runner/work/deepchecks/deepchecks/deepchecks/vision/checks/model_evaluation/model_error_analysis.py:81: DeprecationWarning:
The ModelErrorAnalysis check is deprecated and will be removed in the 0.11 version. Please use the WeakSegmentsPerformance check instead.
Validating Input:
| | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:15]
Validating Input:
|#####| 1/1 [Time: 00:15]
Ingesting Batches - Train Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Train Dataset:
|##5 | 1/2 [Time: 00:09]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:18]
Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:18]
Ingesting Batches - Test Dataset:
| | 0/2 [Time: 00:00]
Ingesting Batches - Test Dataset:
|##5 | 1/2 [Time: 00:08]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:18]
Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:18]
Computing Check:
| | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:92: FutureWarning:
Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:97: FutureWarning:
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Computing Check:
|#####| 1/1 [Time: 00:03]
Computing Check:
|#####| 1/1 [Time: 00:03]
Total running time of the script: ( 1 minutes 5.457 seconds)