Model Error Analysis check#

This notebooks provides an overview for using and understanding the model error analysis check.

Structure:

What is the purpose of the check?#

Imports#

from deepchecks.vision.checks import ModelErrorAnalysis

Classification Performance Report#

Generate data and model:#

from deepchecks.vision.datasets.classification import mnist

mnist_model = mnist.load_model()
train_ds = mnist.load_dataset(train=True, object_type='VisionData')
test_ds = mnist.load_dataset(train=False, object_type='VisionData')

Run the check:#

check = ModelErrorAnalysis(min_error_model_score=-0.1)
result = check.run(train_ds, test_ds, mnist_model)
result
Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:00]

Ingesting Batches - Train Dataset:
|                                                                                                                                                             | 0/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###########                                                                                                                                                  | 11/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|######################                                                                                                                                       | 22/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|#################################                                                                                                                            | 33/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|############################################                                                                                                                 | 44/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|#######################################################                                                                                                      | 55/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|##################################################################                                                                                           | 66/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|#############################################################################                                                                                | 77/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|########################################################################################                                                                     | 88/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|###################################################################################################                                                          | 99/157 [Time: 00:00]

Ingesting Batches - Train Dataset:
|##############################################################################################################                                               | 110/157 [Time: 00:01]

Ingesting Batches - Train Dataset:
|#########################################################################################################################                                    | 121/157 [Time: 00:01]

Ingesting Batches - Train Dataset:
|####################################################################################################################################                         | 132/157 [Time: 00:01]

Ingesting Batches - Train Dataset:
|###############################################################################################################################################              | 143/157 [Time: 00:01]

Ingesting Batches - Train Dataset:
|##########################################################################################################################################################   | 154/157 [Time: 00:01]

Ingesting Batches - Train Dataset:
|#############################################################################################################################################################| 157/157 [Time: 00:01]


Ingesting Batches - Test Dataset:
|          | 0/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|#         | 1/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##        | 2/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|###       | 3/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|####      | 4/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|#####     | 5/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|######    | 6/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|#######   | 7/10 [Time: 00:00]


Ingesting Batches - Test Dataset:
|########  | 8/10 [Time: 00:01]


Ingesting Batches - Test Dataset:
|######### | 9/10 [Time: 00:01]


Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]


Ingesting Batches - Test Dataset:
|##########| 10/10 [Time: 00:01]



Computing Check:
|     | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:92: FutureWarning:

Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327

/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:97: FutureWarning:

Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327





Computing Check:
|#####| 1/1 [Time: 00:04]



Computing Check:
|#####| 1/1 [Time: 00:04]
Model Error Analysis


If you have a GPU, you can speed up this check by passing it as an argument to .run() as device=<your GPU>

To display the results in an IDE like PyCharm, you can use the following code:

#  result.show_in_window()

The result will be displayed in a new window.

Object Detection Class Performance#

For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds.

import numpy as np

from deepchecks.vision.datasets.detection import coco

Generate Data and Model#

We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model

yolo = coco.load_model(pretrained=True)

train_ds = coco.load_dataset(train=True, object_type='VisionData')
test_ds = coco.load_dataset(train=False, object_type='VisionData')

Run the check:#

check = ModelErrorAnalysis(min_error_model_score=-1)
result = check.run(train_ds, test_ds, yolo)
result
Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:10]
Validating Input:
|#####| 1/1 [Time: 00:10]

Ingesting Batches - Train Dataset:
|     | 0/2 [Time: 00:00]

Ingesting Batches - Train Dataset:
|##5  | 1/2 [Time: 00:06]

Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:12]

Ingesting Batches - Train Dataset:
|#####| 2/2 [Time: 00:12]


Ingesting Batches - Test Dataset:
|     | 0/2 [Time: 00:00]


Ingesting Batches - Test Dataset:
|##5  | 1/2 [Time: 00:06]


Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:12]


Ingesting Batches - Test Dataset:
|#####| 2/2 [Time: 00:12]



Computing Check:
|     | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:92: FutureWarning:

Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327

/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:97: FutureWarning:

Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327





Computing Check:
|#####| 1/1 [Time: 00:02]



Computing Check:
|#####| 1/1 [Time: 00:02]
Model Error Analysis


Total running time of the script: ( 0 minutes 45.225 seconds)

Gallery generated by Sphinx-Gallery