Confusion Matrix

This notebooks provides an overview for using and understanding the confusion matrix check.

Structure:

• What is the purpose of the check?

• Generate data & model

• Run the check

What is the purpose of the check?

The confusion matrix check outputs a confusion matrix for both classification problems and object detection problems. In object detection problems, some predictions do not overlap on any label and can be classified as not found in the confusion matrix.

Generate Data and Model

We generate a sample dataset of 128 images from the COCO dataset, and using the YOLOv5 model.

from deepchecks.vision.datasets.detection import coco

yolo = coco.load_model(pretrained=True)
train_ds = coco.load_dataset(object_type='VisionData')

Downloading: "https://github.com/ultralytics/yolov5/archive/v6.1.zip" to /home/runner/.cache/torch/hub/v6.1.zip
Downloading https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s.pt to yolov5s.pt...

  0%|          | 0.00/14.1M [00:00<?, ?B/s]
  0%|          | 48.0k/14.1M [00:00<00:40, 368kB/s]
  1%|1         | 192k/14.1M [00:00<00:18, 802kB/s]
  3%|3         | 448k/14.1M [00:00<00:10, 1.33MB/s]
  6%|6         | 912k/14.1M [00:00<00:06, 2.18MB/s]
 13%|#2        | 1.80M/14.1M [00:00<00:03, 3.97MB/s]
 25%|##5       | 3.58M/14.1M [00:00<00:01, 7.31MB/s]
 30%|###       | 4.27M/14.1M [00:00<00:01, 7.19MB/s]
 52%|#####2    | 7.38M/14.1M [00:01<00:00, 13.4MB/s]
 77%|#######6  | 10.9M/14.1M [00:01<00:00, 17.9MB/s]
100%|##########| 14.1M/14.1M [00:01<00:00, 11.9MB/s]

Run the check

from deepchecks.vision.checks import ConfusionMatrixReport

check = ConfusionMatrixReport(categories_to_display=10)
result = check.run(train_ds, yolo)
result

Validating Input:
|     | 0/1 [Time: 00:00]
Validating Input:
|#####| 1/1 [Time: 00:09]
Validating Input:
|#####| 1/1 [Time: 00:09]

Ingesting Batches:
|     | 0/2 [Time: 00:00]

Ingesting Batches:
|##5  | 1/2 [Time: 00:08]

Ingesting Batches:
|#####| 2/2 [Time: 00:16]

Ingesting Batches:
|#####| 2/2 [Time: 00:16]


Computing Check:
|     | 0/1 [Time: 00:00]


Computing Check:
|#####| 1/1 [Time: 00:00]

If you have a GPU, you can speed up this check by calling:

# check.run(train_ds, yolo, device=<your GPU>)

To display the results in an IDE like PyCharm, you can use the following code:

#  result.show_in_window()

The result will be displayed in a new window.

previous

Class Performance

next

Image Segment Performance