Confusion Matrix

This notebooks provides an overview for using and understanding the confusion matrix check.

Structure:

• What is the purpose of the check?

• Generate data & model

• Run the check

What is the purpose of the check?

The confusion matrix check outputs a confusion matrix for both classification problems and object detection problems. In object detection problems, some predictions do not overlap on any label and can be classified as not found in the confusion matrix.

Generate Data and Model

We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model.

from deepchecks.vision.datasets.detection import coco

yolo = coco.load_model(pretrained=True)
train_ds = coco.load_dataset(object_type='VisionData')

Out:

Downloading https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt to yolov5s.pt...

  0%|          | 0.00/14.1M [00:00<?, ?B/s]
 43%|####2     | 6.02M/14.1M [00:00<00:00, 63.0MB/s]
100%|##########| 14.1M/14.1M [00:00<00:00, 106MB/s]

Run the check

from deepchecks.vision.checks.performance import ConfusionMatrixReport

check = ConfusionMatrixReport(categories_to_display=10)
check.run(train_ds, yolo)

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:05<00:00,  5.35s/ ]


Ingesting Batches:   0%|  | 0/2 [00:00<?, ? Batch/s]
Ingesting Batches:  50%|# | 1/2 [00:05<00:05,  5.40s/ Batch]
Ingesting Batches: 100%|##| 2/2 [00:10<00:00,  5.35s/ Batch]


Computing Check:   0%| | 0/1 [00:00<?, ? Check/s]

Confusion Matrix

Calculate the confusion matrix of the model on the given dataset.

Additional Outputs

Showing 10 of 80 classes:

"No overlapping" categories are labels and prediction which did not have a matching label/prediction.
For example a predictions that did not have a sufficiently overlapping label bounding box will appear under "No overlapping" category in the True Value axis (y-axis).

previous

Class Performance

next

Image Segment Performance