Object Detection Tutorial#

In this tutorial, you will learn how to validate your object detection model using deepchecks test suites. You can read more about the different checks and suites for computer vision use cases at the examples section

If you just want to see the output of this tutorial, jump to Observing the results: section.

An object detection tasks usually consists of two parts:

  • Object Localization, where the model predicts the location of an object in the image,

  • Object Classification, where the model predicts the class of the detected object.

The common output of an object detection model is a list of bounding boxes around the objects, and their classes.

Defining the data and model#

import math
# Importing the required packages
import os
import time
import urllib.request
import xml.etree.ElementTree as ET
import zipfile
from functools import partial

import albumentations as A
import matplotlib.pyplot as plt
import numpy as np
import torch
import torchvision
import torchvision.transforms as T
from albumentations.pytorch import ToTensorV2
from PIL import Image
from torch import nn
from torch.utils.data import DataLoader, Dataset
from torchvision.models.detection import _utils as det_utils
from torchvision.models.detection.ssdlite import SSDLiteClassificationHead

import deepchecks
from deepchecks.vision.detection_data import DetectionData

Load Data#

The model in this tutorial is used to detect tomatoes in images. The model is trained on a dataset consisted of 895 images of tomatoes, with bounding box annotations provided in PASCAL VOC format. All annotations belong to a single class: tomato.

Note

The dataset is available at the following link: https://www.kaggle.com/andrewmvd/tomato-detection

We thank the authors of the dataset for providing the dataset.

url = 'https://figshare.com/ndownloader/files/34488599'
urllib.request.urlretrieve(url, 'tomato-detection.zip')

with zipfile.ZipFile('tomato-detection.zip', 'r') as zip_ref:
    zip_ref.extractall('.')

class TomatoDataset(Dataset):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms

        self.images = list(sorted(os.listdir(os.path.join(root, 'images'))))
        self.annotations = list(sorted(os.listdir(os.path.join(root, 'annotations'))))

    def __getitem__(self, idx):
        img_path = os.path.join(self.root, "images", self.images[idx])
        ann_path = os.path.join(self.root, "annotations", self.annotations[idx])
        img = Image.open(img_path).convert("RGB")
        bboxes = []
        labels = []
        with open(ann_path, 'r') as f:
            tree = ET.parse(f)
            root = tree.getroot()
            size = root.find('size')
            w = int(size.find('width').text)
            h = int(size.find('height').text)

            for obj in root.iter('object'):
                difficult = obj.find('difficult').text
                if int(difficult) == 1:
                    continue
                cls_id = 1
                xmlbox = obj.find('bndbox')
                b = [float(xmlbox.find('xmin').text), float(xmlbox.find('ymin').text), float(xmlbox.find('xmax').text),
                        float(xmlbox.find('ymax').text)]
                bboxes.append(b)
                labels.append(cls_id)

        bboxes = torch.as_tensor(np.array(bboxes), dtype=torch.float32)
        labels = torch.as_tensor(np.array(labels), dtype=torch.int64)

        if self.transforms is not None:
            res = self.transforms(image=np.array(img), bboxes=bboxes, class_labels=labels)

        target = {
            'boxes': [torch.Tensor(x) for x in res['bboxes']],
            'labels': res['class_labels']
        }

        img = res['image']

        return img, target

    def __len__(self):
        return len(self.images)

data_transforms = A.Compose([
    A.Resize(height=256, width=256),
    A.CenterCrop(height=224, width=224),
    A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
    ToTensorV2(),
], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels']))

dataset = TomatoDataset(root=os.path.join(os.path.curdir, 'tomato-detection/data'),
                        transforms=data_transforms)
train_set, val_set = torch.utils.data.random_split(dataset,
                                                    [int(len(dataset)*0.9), len(dataset)-int(len(dataset)*0.9)],
                                                    generator=torch.Generator().manual_seed(42))
val_set.transforms = A.Compose([ToTensorV2()])
train_loader = DataLoader(train_set, batch_size=64, collate_fn=(lambda batch: tuple(zip(*batch))))
val_loader = DataLoader(val_set, batch_size=64, collate_fn=(lambda batch: tuple(zip(*batch))))

Visualize a Few Images#

Let’s visualize a few training images so as to understand the data augmentation.

def prepare(inp):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1) * 255
    inp = inp.transpose((2,0,1))
    return torch.tensor(inp, dtype=torch.uint8)

import torchvision.transforms.functional as F


def show(imgs):
    if not isinstance(imgs, list):
        imgs = [imgs]
    fig, axs = plt.subplots(ncols=len(imgs), squeeze=False, figsize=(20,20))
    for i, img in enumerate(imgs):
        img = img.detach()
        img = F.to_pil_image(img)
        axs[0, i].imshow(np.asarray(img))
        axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])

from torchvision.utils import draw_bounding_boxes

data = next(iter(train_loader))
inp, targets = data[0][:4], data[1][:4]


result = [draw_bounding_boxes(prepare(inp[i]), torch.stack(targets[i]['boxes']),
                              colors=['yellow'] * torch.stack(targets[i]['boxes']).shape[0], width=5)
          for i in range(len(targets))]
show(result)
plot detection tutorialTomatoes with bbox

Downloading a Pre-trained Model#

In this tutorial, we will download a pre-trained SSDlite model and a MobileNetV3 Large backbone from the official PyTorch repository. For more details, please refer to the official documentation.

After downloading the model, we will fine-tune it for our particular classes. We will do it by replacing the pre-trained head with a new one that matches our needs.

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

model = torchvision.models.detection.ssdlite320_mobilenet_v3_large(pretrained=True)

in_channels = det_utils.retrieve_out_channels(model.backbone, (320, 320))
num_anchors = model.anchor_generator.num_anchors_per_location()
norm_layer = partial(nn.BatchNorm2d, eps=0.001, momentum=0.03)

model.head.classification_head = SSDLiteClassificationHead(in_channels, num_anchors, 2, norm_layer)
model.to(device)

Out:

Downloading: "https://download.pytorch.org/models/ssdlite320_mobilenet_v3_large_coco-a79551df.pth" to /home/runner/.cache/torch/hub/checkpoints/ssdlite320_mobilenet_v3_large_coco-a79551df.pth

  0%|          | 0.00/13.4M [00:00<?, ?B/s]
 17%|#7        | 2.33M/13.4M [00:00<00:00, 24.4MB/s]
 36%|###6      | 4.85M/13.4M [00:00<00:00, 23.2MB/s]
 73%|#######3  | 9.84M/13.4M [00:00<00:00, 35.4MB/s]
100%|##########| 13.4M/13.4M [00:00<00:00, 36.0MB/s]

SSD(
  (backbone): SSDLiteFeatureExtractorMobileNet(
    (features): Sequential(
      (0): Sequential(
        (0): ConvNormActivation(
          (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(16, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): Hardswish()
        )
        (1): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(16, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (1): ConvNormActivation(
              (0): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(16, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (2): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (1): ConvNormActivation(
              (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=64, bias=False)
              (1): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (2): ConvNormActivation(
              (0): Conv2d(64, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(24, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (3): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(72, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (1): ConvNormActivation(
              (0): Conv2d(72, 72, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=72, bias=False)
              (1): BatchNorm2d(72, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (2): ConvNormActivation(
              (0): Conv2d(72, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(24, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (4): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(72, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (1): ConvNormActivation(
              (0): Conv2d(72, 72, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), groups=72, bias=False)
              (1): BatchNorm2d(72, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(72, 24, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(24, 72, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(72, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(40, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (5): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(40, 120, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(120, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (1): ConvNormActivation(
              (0): Conv2d(120, 120, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=120, bias=False)
              (1): BatchNorm2d(120, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(120, 32, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(32, 120, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(120, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(40, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (6): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(40, 120, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(120, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (1): ConvNormActivation(
              (0): Conv2d(120, 120, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=120, bias=False)
              (1): BatchNorm2d(120, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): ReLU(inplace=True)
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(120, 32, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(32, 120, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(120, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(40, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (7): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(40, 240, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(240, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(240, 240, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=240, bias=False)
              (1): BatchNorm2d(240, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): ConvNormActivation(
              (0): Conv2d(240, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (8): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(80, 200, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(200, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(200, 200, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=200, bias=False)
              (1): BatchNorm2d(200, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): ConvNormActivation(
              (0): Conv2d(200, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (9): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(80, 184, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(184, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(184, 184, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=184, bias=False)
              (1): BatchNorm2d(184, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): ConvNormActivation(
              (0): Conv2d(184, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (10): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(80, 184, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(184, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(184, 184, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=184, bias=False)
              (1): BatchNorm2d(184, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): ConvNormActivation(
              (0): Conv2d(184, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (11): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(480, 480, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=480, bias=False)
              (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(480, 120, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(120, 480, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(480, 112, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(112, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (12): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(672, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=672, bias=False)
              (1): BatchNorm2d(672, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(672, 168, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(168, 672, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(672, 112, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(112, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (13): ConvNormActivation(
          (0): Conv2d(112, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(672, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): Hardswish()
        )
      )
      (1): Sequential(
        (0): Sequential(
          (1): ConvNormActivation(
            (0): Conv2d(672, 672, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), groups=672, bias=False)
            (1): BatchNorm2d(672, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): Hardswish()
          )
          (2): SqueezeExcitation(
            (avgpool): AdaptiveAvgPool2d(output_size=1)
            (fc1): Conv2d(672, 168, kernel_size=(1, 1), stride=(1, 1))
            (fc2): Conv2d(168, 672, kernel_size=(1, 1), stride=(1, 1))
            (activation): ReLU()
            (scale_activation): Hardsigmoid()
          )
          (3): ConvNormActivation(
            (0): Conv2d(672, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          )
        )
        (1): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(480, 480, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=480, bias=False)
              (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(480, 120, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(120, 480, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(480, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (2): InvertedResidual(
          (block): Sequential(
            (0): ConvNormActivation(
              (0): Conv2d(80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (1): ConvNormActivation(
              (0): Conv2d(480, 480, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=480, bias=False)
              (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
              (2): Hardswish()
            )
            (2): SqueezeExcitation(
              (avgpool): AdaptiveAvgPool2d(output_size=1)
              (fc1): Conv2d(480, 120, kernel_size=(1, 1), stride=(1, 1))
              (fc2): Conv2d(120, 480, kernel_size=(1, 1), stride=(1, 1))
              (activation): ReLU()
              (scale_activation): Hardsigmoid()
            )
            (3): ConvNormActivation(
              (0): Conv2d(480, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(80, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            )
          )
        )
        (3): ConvNormActivation(
          (0): Conv2d(80, 480, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): Hardswish()
        )
      )
    )
    (extra): ModuleList(
      (0): Sequential(
        (0): ConvNormActivation(
          (0): Conv2d(480, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvNormActivation(
          (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=256, bias=False)
          (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): ConvNormActivation(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
      )
      (1): Sequential(
        (0): ConvNormActivation(
          (0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvNormActivation(
          (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=128, bias=False)
          (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): ConvNormActivation(
          (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
      )
      (2): Sequential(
        (0): ConvNormActivation(
          (0): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvNormActivation(
          (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=128, bias=False)
          (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): ConvNormActivation(
          (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
      )
      (3): Sequential(
        (0): ConvNormActivation(
          (0): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvNormActivation(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=64, bias=False)
          (1): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): ConvNormActivation(
          (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
      )
    )
  )
  (anchor_generator): DefaultBoxGenerator(aspect_ratios=[[2, 3], [2, 3], [2, 3], [2, 3], [2, 3], [2, 3]], clip=True, scales=[0.2, 0.35, 0.5, 0.65, 0.8, 0.95, 1.0], steps=None)
  (head): SSDLiteHead(
    (classification_head): SSDLiteClassificationHead(
      (module_list): ModuleList(
        (0): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=672, bias=False)
            (1): BatchNorm2d(672, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(672, 12, kernel_size=(1, 1), stride=(1, 1))
        )
        (1): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(480, 480, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=480, bias=False)
            (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(480, 12, kernel_size=(1, 1), stride=(1, 1))
        )
        (2): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512, bias=False)
            (1): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1))
        )
        (3): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(256, 12, kernel_size=(1, 1), stride=(1, 1))
        )
        (4): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(256, 12, kernel_size=(1, 1), stride=(1, 1))
        )
        (5): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=128, bias=False)
            (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(128, 12, kernel_size=(1, 1), stride=(1, 1))
        )
      )
    )
    (regression_head): SSDLiteRegressionHead(
      (module_list): ModuleList(
        (0): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=672, bias=False)
            (1): BatchNorm2d(672, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(672, 24, kernel_size=(1, 1), stride=(1, 1))
        )
        (1): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(480, 480, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=480, bias=False)
            (1): BatchNorm2d(480, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(480, 24, kernel_size=(1, 1), stride=(1, 1))
        )
        (2): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512, bias=False)
            (1): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(512, 24, kernel_size=(1, 1), stride=(1, 1))
        )
        (3): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(256, 24, kernel_size=(1, 1), stride=(1, 1))
        )
        (4): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (1): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(256, 24, kernel_size=(1, 1), stride=(1, 1))
        )
        (5): Sequential(
          (0): ConvNormActivation(
            (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=128, bias=False)
            (1): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
            (2): ReLU6(inplace=True)
          )
          (1): Conv2d(128, 24, kernel_size=(1, 1), stride=(1, 1))
        )
      )
    )
  )
  (transform): GeneralizedRCNNTransform(
      Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
      Resize(min_size=(320,), max_size=320, mode='bilinear')
  )
)

Loading Pre-trained Weights#

For this tutorial we will not include the training code itself, but will download and load pre-trained weights.

model.load_state_dict(torch.load('tomato-detection/ssd_model.pth'))
_ = model.eval()

Validating the Model With Deepchecks#

Now, after we have the training data, validation data and the model, we can validate the model with deepchecks test suites.

Visualize the Data Loader and the Model Outputs#

First we’ll make sure we are familiar with the data loader and the model outputs.

batch = next(iter(train_loader))

print("Batch type is: ", type(batch))
print("First element is: ", type(batch[0]), "with len of ", len(batch[0]))
print("Example output of an image shape from the dataloader ", batch[0][0].shape)
print("Image values", batch[0][0])
print("-"*80)

print("Second element is: ", type(batch[1]), "with len of ", len(batch[1]))
print("Example output of a label shape from the dataloader ", batch[1][0])

Out:

Batch type is:  <class 'tuple'>
First element is:  <class 'tuple'> with len of  64
Example output of an image shape from the dataloader  torch.Size([3, 224, 224])
Image values tensor([[[-1.79253, -1.82678, -1.82678,  ...,  1.39267,  1.34130,  1.32417],
         [-1.72403, -1.79253, -1.80966,  ...,  1.35842,  1.32417,  1.34130],
         [-1.75828, -1.74116, -1.70691,  ...,  1.32417,  1.34130,  1.35842],
         ...,
         [-1.84391, -1.82678, -1.75828,  ...,  0.62206,  0.19394, -0.35405],
         [-1.80966, -1.79253, -1.72403,  ...,  0.81043,  0.72481,  0.34806],
         [-1.79253, -1.84391, -1.75828,  ...,  0.81043,  0.82755,  0.69056]],

        [[-1.38796, -1.45798, -1.45798,  ...,  1.51821,  1.46569,  1.46569],
         [-1.38796, -1.47549, -1.52801,  ...,  1.50070,  1.46569,  1.48319],
         [-1.42297, -1.47549, -1.49300,  ...,  1.46569,  1.50070,  1.50070],
         ...,
         [-1.70308, -1.68557, -1.61555,  ...,  0.67787,  0.22269, -0.33753],
         [-1.68557, -1.66807, -1.58053,  ...,  0.87045,  0.74790,  0.38025],
         [-1.68557, -1.70308, -1.61555,  ...,  0.87045,  0.85294,  0.71289]],

        [[-1.57786, -1.61272, -1.61272,  ...,  1.66397,  1.61168,  1.59425],
         [-1.54301, -1.59529, -1.64758,  ...,  1.62911,  1.59425,  1.59425],
         [-1.59529, -1.59529, -1.61272,  ...,  1.59425,  1.61168,  1.62911],
         ...,
         [-1.63015, -1.59529, -1.52558,  ...,  0.46135,  0.02562, -0.53211],
         [-1.59529, -1.54301, -1.47329,  ...,  0.72279,  0.61821,  0.21734],
         [-1.59529, -1.59529, -1.50815,  ...,  0.72279,  0.72279,  0.56593]]])
--------------------------------------------------------------------------------
Second element is:  <class 'tuple'> with len of  64
Example output of a label shape from the dataloader  {'boxes': [tensor([  0.00000,  75.13600,  39.68000, 165.75999]), tensor([ 0.00000,  0.00000, 94.08000, 93.56800])], 'labels': [tensor(1), tensor(1)]}

Implementing the DetectionData class#

The checks in the package validate the model & data by calculating various quantities over the data, labels and predictions. In order to do that, those must be in a pre-defined format, according to the task type. The first step is to implement a class that enables deepchecks to interact with your model and data and transform them to this pre-defined format, which is set for each task type. In this tutorial, we will implement the object detection task type by implementing a class that inherits from the deepchecks.vision.detection_data.DetectionData class.

The DetectionData class contains additional data and general methods intended for easy access to relevant metadata for object detection ML models validation. To learn more about the expected format please visit the API reference for the deepchecks.vision.detection_data.DetectionData class.

from deepchecks.vision.detection_data import DetectionData


class TomatoData(DetectionData):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def batch_to_images(self, batch):
        """
        Convert a batch of data to images in the expected format. The expected format is an iterable of cv2 images,
        where each image is a numpy array of shape (height, width, channels). The numbers in the array should be in the
        range [0, 255] in a uint8 format.
        """
        inp = torch.stack(list(batch[0])).numpy().transpose((0, 2, 3, 1))
        mean = [0.485, 0.456, 0.406]
        std = [0.229, 0.224, 0.225]
        # Un-normalize the images
        inp = std * inp + mean
        inp = np.clip(inp, 0, 1)
        return inp * 255

    def batch_to_labels(self, batch):
        """
        Convert a batch of data to labels in the expected format. The expected format is a list of tensors of length N,
        where N is the number of samples. Each tensor element is in a shape of [B, 5], where B is the number of bboxes
        in the image, and each bounding box is in the structure of [class_id, x, y, w, h].
        """
        tensor_annotations = batch[1]
        label = []
        for annotation in tensor_annotations:
            if len(annotation["boxes"]):
                bbox = torch.stack(annotation["boxes"])
                # Convert the Pascal VOC xyxy format to xywh format
                bbox[:, 2:] = bbox[:, 2:] - bbox[:, :2]
                # The label shape is [class_id, x, y, w, h]
                label.append(
                    torch.concat([torch.stack(annotation["labels"]).reshape((-1, 1)), bbox], dim=1)
                )
            else:
                # If it's an empty image, we need to add an empty label
                label.append(torch.tensor([]))
        return label

    def infer_on_batch(self, batch, model, device):
        """
        Returns the predictions for a batch of data. The expected format is a list of tensors of shape length N, where N
        is the number of samples. Each tensor element is in a shape of [B, 6], where B is the number of bboxes in the
        predictions, and each bounding box is in the structure of [x, y, w, h, score, class_id].
        """
        nm_thrs = 0.2
        score_thrs = 0.7
        imgs = list(img.to(device) for img in batch[0])
        # Getting the predictions of the model on the batch
        with torch.no_grad():
            preds = model(imgs)
        processed_pred = []
        for pred in preds:
            # Performoing non-maximum suppression on the detections
            keep_boxes = torchvision.ops.nms(pred['boxes'], pred['scores'], nm_thrs)
            score_filter = pred['scores'][keep_boxes] > score_thrs

            # get the filtered result
            test_boxes = pred['boxes'][keep_boxes][score_filter].reshape((-1, 4))
            test_boxes[:, 2:] = test_boxes[:, 2:] - test_boxes[:, :2]  # xyxy to xywh
            test_labels = pred['labels'][keep_boxes][score_filter]
            test_scores = pred['scores'][keep_boxes][score_filter]

            processed_pred.append(
                torch.concat([test_boxes, test_scores.reshape((-1, 1)), test_labels.reshape((-1, 1))], dim=1))
        return processed_pred

After defining the task class, we can validate it by running the following code:

# We have a single label here, which is the tomato class
# The label_map is a dictionary that maps the class id to the class name, for display purposes.
LABEL_MAP = {
    1: 'Tomato'
}
training_data = TomatoData(data_loader=train_loader, label_map=LABEL_MAP)
val_data = TomatoData(data_loader=val_loader, label_map=LABEL_MAP)

training_data.validate_format(model, device=device)
val_data.validate_format(model, device=device)

# And observe the output:

Out:

Deepchecks will try to validate the extractors given...
torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
Structure validation
--------------------
Label formatter: Pass!
Prediction formatter: Pass!
Image formatter: Pass!

Content validation
------------------
For validating the content within the structure you have to manually observe the classes, image, label and prediction.
Examples of classes observed in the batch's labels: [[1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1], [1]]
Visual images & label & prediction: should open in a new window
*******************************************************************************
This machine does not support GUI
The formatted image was saved in:
/home/runner/work/deepchecks/deepchecks/docs/source/user-guide/vision/tutorials/deepchecks_formatted_image (4).jpg
Visual examples of an image with prediction and label data. Label is red, prediction is blue, and deepchecks loves you.
validate_extractors can be set to skip the image saving or change the save path
*******************************************************************************
Deepchecks will try to validate the extractors given...
Structure validation
--------------------
Label formatter: Pass!
Prediction formatter: Pass!
Image formatter: Pass!

Content validation
------------------
For validating the content within the structure you have to manually observe the classes, image, label and prediction.
Examples of classes observed in the batch's labels: [[1, 1, 1, 1], [1], [1, 1, 1, 1, 1, 1, 1], [1], [1]]
Visual images & label & prediction: should open in a new window
*******************************************************************************
This machine does not support GUI
The formatted image was saved in:
/home/runner/work/deepchecks/deepchecks/docs/source/user-guide/vision/tutorials/deepchecks_formatted_image (5).jpg
Visual examples of an image with prediction and label data. Label is red, prediction is blue, and deepchecks loves you.
validate_extractors can be set to skip the image saving or change the save path
*******************************************************************************

Running Deepchecks’ full suite on our data and model!#

Now that we have defined the task class, we can validate the model with the full suite of deepchecks. This can be done with this simple few lines of code:

from deepchecks.vision.suites import full_suite

suite = full_suite()
result = suite.run(training_data, val_data, model, device=device)

Out:

Validating Input:   0%| | 0/1 [00:00<?, ? /s]
Validating Input: 100%|#| 1/1 [00:06<00:00,  6.62s/ ]

Ingesting Batches - Train Dataset:   0%|             | 0/13 [00:00<?, ? Batch/s]

Ingesting Batches - Train Dataset:   8%|#            | 1/13 [00:04<00:59,  4.94s/ Batch]

Ingesting Batches - Train Dataset:  15%|##           | 2/13 [00:09<00:54,  4.97s/ Batch]

Ingesting Batches - Train Dataset:  23%|###          | 3/13 [00:15<00:51,  5.12s/ Batch]

Ingesting Batches - Train Dataset:  31%|####         | 4/13 [00:20<00:45,  5.10s/ Batch]

Ingesting Batches - Train Dataset:  38%|#####        | 5/13 [00:25<00:41,  5.16s/ Batch]

Ingesting Batches - Train Dataset:  46%|######       | 6/13 [00:30<00:36,  5.22s/ Batch]

Ingesting Batches - Train Dataset:  54%|#######      | 7/13 [00:35<00:30,  5.16s/ Batch]

Ingesting Batches - Train Dataset:  62%|########     | 8/13 [00:41<00:25,  5.14s/ Batch]

Ingesting Batches - Train Dataset:  69%|#########    | 9/13 [00:46<00:20,  5.14s/ Batch]

Ingesting Batches - Train Dataset:  77%|##########   | 10/13 [00:51<00:15,  5.19s/ Batch]

Ingesting Batches - Train Dataset:  85%|###########  | 11/13 [00:56<00:10,  5.14s/ Batch]

Ingesting Batches - Train Dataset:  92%|############ | 12/13 [01:01<00:05,  5.17s/ Batch]

Ingesting Batches - Train Dataset: 100%|#############| 13/13 [01:04<00:00,  4.46s/ Batch]


Computing Single Dataset Checks - Train Dataset:   0%|      | 0/6 [00:00<?, ? Check/s]


Computing Single Dataset Checks - Train Dataset:   0%|      | 0/6 [00:00<?, ? Check/s, Check=Mean Average Precision Report]


Computing Single Dataset Checks - Train Dataset:  17%|#     | 1/6 [00:00<00:01,  4.30 Check/s, Check=Mean Average Precision Report]


Computing Single Dataset Checks - Train Dataset:  17%|#     | 1/6 [00:00<00:01,  4.30 Check/s, Check=Mean Average Recall Report]


Computing Single Dataset Checks - Train Dataset:  33%|##    | 2/6 [00:00<00:00,  4.77 Check/s, Check=Mean Average Recall Report]


Computing Single Dataset Checks - Train Dataset:  33%|##    | 2/6 [00:00<00:00,  4.77 Check/s, Check=Confusion Matrix Report]


Computing Single Dataset Checks - Train Dataset:  50%|###   | 3/6 [00:00<00:00,  4.77 Check/s, Check=Image Segment Performance]


Computing Single Dataset Checks - Train Dataset:  67%|####  | 4/6 [00:06<00:04,  2.12s/ Check, Check=Image Segment Performance]


Computing Single Dataset Checks - Train Dataset:  67%|####  | 4/6 [00:06<00:04,  2.12s/ Check, Check=Image Property Outliers]


Computing Single Dataset Checks - Train Dataset:  83%|##### | 5/6 [00:07<00:01,  1.56s/ Check, Check=Image Property Outliers]


Computing Single Dataset Checks - Train Dataset:  83%|##### | 5/6 [00:07<00:01,  1.56s/ Check, Check=Label Property Outliers]


Computing Single Dataset Checks - Train Dataset: 100%|######| 6/6 [00:07<00:00,  1.20s/ Check, Check=Label Property Outliers]



Ingesting Batches - Test Dataset:   0%|  | 0/2 [00:00<?, ? Batch/s]



Ingesting Batches - Test Dataset:  50%|# | 1/2 [00:05<00:05,  5.08s/ Batch]



Ingesting Batches - Test Dataset: 100%|##| 2/2 [00:06<00:00,  3.22s/ Batch]




Computing Single Dataset Checks - Test Dataset:   0%|      | 0/6 [00:00<?, ? Check/s]




Computing Single Dataset Checks - Test Dataset:   0%|      | 0/6 [00:00<?, ? Check/s, Check=Mean Average Precision Report]




Computing Single Dataset Checks - Test Dataset:  17%|#     | 1/6 [00:00<00:00, 13.05 Check/s, Check=Mean Average Recall Report]




Computing Single Dataset Checks - Test Dataset:  33%|##    | 2/6 [00:00<00:00, 17.47 Check/s, Check=Mean Average Recall Report]




Computing Single Dataset Checks - Test Dataset:  33%|##    | 2/6 [00:00<00:00, 17.47 Check/s, Check=Confusion Matrix Report]




Computing Single Dataset Checks - Test Dataset:  50%|###   | 3/6 [00:00<00:00, 17.47 Check/s, Check=Image Segment Performance]




Computing Single Dataset Checks - Test Dataset:  67%|####  | 4/6 [00:00<00:00,  3.64 Check/s, Check=Image Segment Performance]




Computing Single Dataset Checks - Test Dataset:  67%|####  | 4/6 [00:00<00:00,  3.64 Check/s, Check=Image Property Outliers]




Computing Single Dataset Checks - Test Dataset:  83%|##### | 5/6 [00:01<00:00,  4.24 Check/s, Check=Image Property Outliers]




Computing Single Dataset Checks - Test Dataset:  83%|##### | 5/6 [00:01<00:00,  4.24 Check/s, Check=Label Property Outliers]




Computing Single Dataset Checks - Test Dataset: 100%|######| 6/6 [00:01<00:00,  3.91 Check/s, Check=Label Property Outliers]





Computing Checks:   0%|           | 0/11 [00:00<?, ? Check/s]





Computing Checks:   0%|           | 0/11 [00:00<?, ? Check/s, Check=Class Performance]





Computing Checks:   9%|#          | 1/11 [00:00<00:01,  8.00 Check/s, Check=Class Performance]





Computing Checks:   9%|#          | 1/11 [00:00<00:01,  8.00 Check/s, Check=Simple Model Comparison]





Computing Checks:  18%|##         | 2/11 [00:00<00:01,  8.00 Check/s, Check=Model Error Analysis]





Computing Checks:  27%|###        | 3/11 [00:00<00:00, 10.68 Check/s, Check=Model Error Analysis]





Computing Checks:  27%|###        | 3/11 [00:00<00:00, 10.68 Check/s, Check=Similar Image Leakage]





Computing Checks:  36%|####       | 4/11 [00:00<00:00, 10.68 Check/s, Check=Heatmap Comparison]





Computing Checks:  45%|#####      | 5/11 [00:00<00:00,  7.82 Check/s, Check=Heatmap Comparison]





Computing Checks:  45%|#####      | 5/11 [00:00<00:00,  7.82 Check/s, Check=Train Test Label Drift]





Computing Checks:  55%|######     | 6/11 [00:00<00:00,  7.82 Check/s, Check=Train Test Label Drift]





Computing Checks:  55%|######     | 6/11 [00:00<00:00,  7.82 Check/s, Check=Train Test Prediction Drift]





Computing Checks:  64%|#######    | 7/11 [00:00<00:00,  8.26 Check/s, Check=Train Test Prediction Drift]





Computing Checks:  64%|#######    | 7/11 [00:00<00:00,  8.26 Check/s, Check=Image Property Drift]





Computing Checks:  73%|########   | 8/11 [00:01<00:00,  6.66 Check/s, Check=Image Property Drift]





Computing Checks:  73%|########   | 8/11 [00:01<00:00,  6.66 Check/s, Check=Image Dataset Drift] Calculating permutation feature importance. Expected to finish in 1 seconds






Computing Checks:  82%|#########  | 9/11 [00:01<00:00,  6.89 Check/s, Check=Image Dataset Drift]





Computing Checks:  82%|#########  | 9/11 [00:01<00:00,  6.89 Check/s, Check=Simple Feature Contribution]





Computing Checks:  91%|########## | 10/11 [00:01<00:00,  6.89 Check/s, Check=New Labels]

Observing the results:#

The results can be saved as a html file with the following code:

result.save_as_html('output.html')

Out:

'output (2).html'

Or, if working inside a notebook, the output can be displayed directly by simply printing the result object:

result

Full Suite

The suite is composed of various checks such as: Image Dataset Drift, Similar Image Leakage, Image Property Drift, etc...
Each check may contain conditions (which will result in pass / fail / warning / error , represented by / / ! / ) as well as other outputs such as plots or tables.
Suites, checks and conditions can all be modified. Read more about custom suites.


Conditions Summary

Status Check Condition More Info
Class Performance Train-Test scores relative degradation is not greater than 0.1 Average Recall for class Tomato (train=0.06 test=0.04) Average Precision for class Tomato (train=0.03 test=0.01)
Mean Average Precision Report - Test Dataset mAP score is not less than 0.3 mAP score is: 0.01
Mean Average Precision Report - Train Dataset mAP score is not less than 0.3 mAP score is: 0.03
Image Segment Performance - Test Dataset No segment with ratio between score to mean less than 80% Properties with failed segments: Brightness: {'Range': '(-inf, 0.33)', 'Metric': 'Average Precision', 'Ratio': 0.11}, Mean Blue Relative Intensity: {'Range': '[0.2, 0.23)', 'Metric': 'Average Precision', 'Ratio': 0.29}, Mean Green Relative Intensity: {'Range': '[0.41, 0.45)', 'Metric': 'Average Reca...
Image Segment Performance - Train Dataset No segment with ratio between score to mean less than 80% Properties with failed segments: Brightness: {'Range': '[0.32, 0.38)', 'Metric': 'Average Recall', 'Ratio': 0.53}, Mean Blue Relative Intensity: {'Range': '(-inf, 0.2)', 'Metric': 'Average Recall', 'Ratio': 0.68}, Mean Green Relative Intensity: {'Range': '[0.45, 0.48)', 'Metric': 'Average Precision'...
Train Test Prediction Drift PSI <= 0.15 and Earth Mover's Distance <= 0.075 for prediction drift
Image Property Drift Earth Mover's Distance <= 0.1 for image properties drift
Simple Feature Contribution Train-Test properties' Predictive Power Score difference is not greater than 0.2
New Labels Percentage of new labels in the test set not above 0.5%.
Similar Image Leakage Number of similar images between train and test is not greater than 0
Train Test Label Drift PSI <= 0.15 and Earth Mover's Distance <= 0.075 for label drift

Check With Conditions Output

Class Performance

Summarize given metrics on a dataset and model.

Conditions Summary
Status Condition More Info
Train-Test scores relative degradation is not greater than 0.1 Average Recall for class Tomato (train=0.06 test=0.04) Average Precision for class Tomato (train=0.03 test=0.01)
Additional Outputs

Go to top

Mean Average Precision Report - Test Dataset

Summarize mean average precision metrics on a dataset and model per IoU and bounding box area.

Conditions Summary
Status Condition More Info
mAP score is not less than 0.3 mAP score is: 0.01
Additional Outputs
  mAP@[.50::.95] (avg.%) mAP@.50 (%) mAP@.75 (%)
Area size      
All 0.01 0.05 0.01
Small (area < 32^2) 0.00 0.01 0.00
Medium (32^2 < area < 96^2) 0.06 0.20 0.02
Large (area < 96^2) 0.00 0.00 0.00

Go to top

Mean Average Precision Report - Train Dataset

Summarize mean average precision metrics on a dataset and model per IoU and bounding box area.

Conditions Summary
Status Condition More Info
mAP score is not less than 0.3 mAP score is: 0.03
Additional Outputs
  mAP@[.50::.95] (avg.%) mAP@.50 (%) mAP@.75 (%)
Area size      
All 0.03 0.07 0.02
Small (area < 32^2) 0.00 0.00 0.00
Medium (32^2 < area < 96^2) 0.10 0.27 0.06
Large (area < 96^2) 0.02 0.04 0.02

Go to top

Train Test Prediction Drift

Calculate prediction drift between train dataset and test dataset, using statistical measures.

Conditions Summary
Status Condition More Info
PSI <= 0.15 and Earth Mover's Distance <= 0.075 for prediction drift
Additional Outputs
The Drift score is a measure for the difference between two distributions. In this check, drift is measured for the distribution of the following prediction properties: ['Samples Per Class', 'Bounding Box Area (in pixels)', 'Number of Bounding Boxes Per Image'].

Go to top

Image Property Drift

Calculate drift between train dataset and test dataset per image property, using statistical measures.

Conditions Summary
Status Condition More Info
Earth Mover's Distance <= 0.1 for image properties drift
Additional Outputs
The Drift score is a measure for the difference between two distributions. In this check, drift is measured for the distribution of the following image properties: ['Area', 'Aspect Ratio', 'Brightness', 'Mean Blue Relative Intensity', 'Mean Green Relative Intensity', 'Mean Red Relative Intensity', 'RMS Contrast'].

Go to top

Image Segment Performance - Test Dataset

Segment the data by various properties of the image, and compare the performance of the segments.

Conditions Summary
Status Condition More Info
No segment with ratio between score to mean less than 80% Properties with failed segments: Brightness: {'Range': '(-inf, 0.33)', 'Metric': 'Average Precision', 'Ratio': 0.11}, Mean Blue Relative Intensity: {'Range': '[0.2, 0.23)', 'Metric': 'Average Precision', 'Ratio': 0.29}, Mean Green Relative Intensity: {'Range': '[0.41, 0.45)', 'Metric': 'Average Recall', 'Ratio': 0.28}, Mean Red Relative Intensity: {'Range': '(-inf, 0.31)', 'Metric': 'Average Precision', 'Ratio': 0.03}, RMS Contrast: {'Range': '[0.16, 0.19)', 'Metric': 'Average Precision', 'Ratio': 0.45}
Additional Outputs

Go to top

Image Segment Performance - Train Dataset

Segment the data by various properties of the image, and compare the performance of the segments.

Conditions Summary
Status Condition More Info
No segment with ratio between score to mean less than 80% Properties with failed segments: Brightness: {'Range': '[0.32, 0.38)', 'Metric': 'Average Recall', 'Ratio': 0.53}, Mean Blue Relative Intensity: {'Range': '(-inf, 0.2)', 'Metric': 'Average Recall', 'Ratio': 0.68}, Mean Green Relative Intensity: {'Range': '[0.45, 0.48)', 'Metric': 'Average Precision', 'Ratio': 0.54}, Mean Red Relative Intensity: {'Range': '(-inf, 0.31)', 'Metric': 'Average Recall', 'Ratio': 0.71}, RMS Contrast: {'Range': '[0.17, 0.2)', 'Metric': 'Average Precision', 'Ratio': 0.64}
Additional Outputs

Go to top

Train Test Label Drift

Calculate label drift between train dataset and test dataset, using statistical measures.

Conditions Summary
Status Condition More Info
PSI <= 0.15 and Earth Mover's Distance <= 0.075 for label drift
Additional Outputs
The Drift score is a measure for the difference between two distributions. In this check, drift is measured for the distribution of the following label properties: ['Samples Per Class', 'Bounding Box Area (in pixels)', 'Number of Bounding Boxes Per Image'].

Go to top

Check Without Conditions Output

Image Dataset Drift

Calculate drift between the entire train and test datasets (based on image properties) using a trained model.

Additional Outputs
The shown features are the image properties (brightness, aspect ratio, etc.) that are most important for the domain classifier - the domain_classifier trained to distinguish between the train and test datasets.
The percents of explained dataset difference are the importance values for the feature calculated using `permutation_importance`.

Main features contributing to drift

* showing only the top 3 columns, you can change it using n_top_columns param

Go to top

Image Property Outliers - Test Dataset

Find outliers images with respect to the given properties.

Additional Outputs

Property "Aspect Ratio"

No outliers found.

Property "Area"

No outliers found.

Property "Brightness"

No outliers found.

Property "RMS Contrast"

Total number of outliers: 1
Non-outliers range: 0.06 to 0.34
RMS Contrast
0.34
Image

Property "Mean Red Relative Intensity"

Total number of outliers: 4
Non-outliers range: 0.28 to 0.4
Mean Red Relative Intensity
0.41
0.41
0.42
0.44
Image

Property "Mean Green Relative Intensity"

No outliers found.

Property "Mean Blue Relative Intensity"

No outliers found.

Go to top

Image Property Outliers - Train Dataset

Find outliers images with respect to the given properties.

Additional Outputs

Property "Aspect Ratio"

No outliers found.

Property "Area"

No outliers found.

Property "Brightness"

Total number of outliers: 12
Non-outliers range: 0.18 to 0.73
Brightness
0.76
0.76
0.78
0.84
0.86
Image