.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "user-guide/vision/auto_tutorials/plot_extractors_validating.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_user-guide_vision_auto_tutorials_plot_extractors_validating.py: Test Your Deepchecks Vision Data Class ================================================ "Data Classes" are used to transform the structure of your data to the structure required for deepchecks. To help ensure they work as intended, deepchecks has built-in helper functions for validating their structure. This guide will demonstrate how to use this helper function to implement them for your own data, step by step. **Structure:** * `Load data and model <#load-data-and-model>`__ * `Create simple DetectionData object <#create-simple-detectiondata-object>`__ * `Running the extractors validation <#running-the-extractors-validation>`__ * `Understand validation results <#understand-validation-results>`__ * `The end result <#the-end-result>`__ .. GENERATED FROM PYTHON SOURCE LINES 22-25 Load data and model ------------------- In the first step we load the DataLoader and our model .. GENERATED FROM PYTHON SOURCE LINES 25-31 .. code-block:: default from deepchecks.vision.datasets.detection.coco import load_dataset, load_model data_loader = load_dataset(train=False, batch_size=1000, object_type='DataLoader') model = load_model() .. rst-class:: sphx-glr-script-out .. code-block:: none Downloading https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt to yolov5s.pt... 0%| | 0.00/14.1M [00:00`__. .. GENERATED FROM PYTHON SOURCE LINES 62-68 .. code-block:: default from deepchecks.vision.utils.validation import validate_extractors validate_extractors(CocoDetectionData(data_loader), model) .. rst-class:: sphx-glr-script-out .. code-block:: none Deepchecks will try to validate the extractors given... Structure validation -------------------- Label formatter: Pass! Prediction formatter: Check requires detection predictions to be a sequence with an entry for each sample Image formatter: Fail! The data inside the iterable must be a numpy array. Content validation ------------------ For validating the content within the structure you have to manually observe the classes, image, label and prediction. Examples of classes observed in the batch's labels: [[597, 253, 107, 26, 76, 247, 193, 149, 0, 545, 223, 218, 20, 174, 19, 512], [431, 181, 112, 215, 219, 563, 209, 546, 35, 224], [76, 73], [180, 2], [60, 0]] Visual images & label & prediction: Unable to show due to invalid image formatter. .. GENERATED FROM PYTHON SOURCE LINES 69-84 Understand validation results ----------------------------- When looking at the result we can see is that it is separated into 2 parts. First part is about the structure we expect to get. This validation is automatic since it's purely technical and doesn't check content correctness. For example, in our validation above we see that the label extractor is passing, meaning the labels are provided in the expected format. Second part is about the content, which cannot be automatically validated and requires your attention. This part includes looking visually at data outputted by the formatters to validate it is correct. In the validation above we see a list of classes that doesn't seem to make much sense - it contains class_ids in values ranging from 0 to 596 while in the COCO dataset there are only 80 classes. For the next step we'll fix the label extractor and then validate again: .. GENERATED FROM PYTHON SOURCE LINES 84-106 .. code-block:: default class CocoDetectionData(DetectionData): def batch_to_labels(self, batch): # Translate labels to deepchecks format. # Originally the label_id was at the last position of the tensor while Deepchecks expects it # to be at the first position. formatted_labels = [] for tensor in batch[1]: tensor = torch.index_select(tensor, 1, torch.LongTensor([4, 0, 1, 2, 3])) if len(tensor) > 0 else tensor formatted_labels.append(tensor) return formatted_labels def batch_to_images(self, batch): return batch[0] def infer_on_batch(self, batch, model, device): return model.to(device)(batch[0]) validate_extractors(CocoDetectionData(data_loader), model) .. rst-class:: sphx-glr-script-out .. code-block:: none Deepchecks will try to validate the extractors given... Structure validation -------------------- Label formatter: Pass! Prediction formatter: Check requires detection predictions to be a sequence with an entry for each sample Image formatter: Fail! The data inside the iterable must be a numpy array. Content validation ------------------ For validating the content within the structure you have to manually observe the classes, image, label and prediction. Examples of classes observed in the batch's labels: [[32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 35, 0, 0, 0, 0], [32, 0, 0, 34, 35, 35, 0, 0, 0, 0], [71, 45], [15, 2], [38, 0]] Visual images & label & prediction: Unable to show due to invalid image formatter. .. GENERATED FROM PYTHON SOURCE LINES 107-109 Now we can see in the content section that our classes are indeed as we expect them to be, values between 0 and 79. Now we can continue and fix the prediction extractor .. GENERATED FROM PYTHON SOURCE LINES 109-133 .. code-block:: default class CocoDetectionData(DetectionData): def infer_on_batch(self, batch, model, device): # Convert from yolo Detections object to List (per image) of Tensors of the shape [B, 6]""" return_list = [] predictions = model.to(device)(batch[0]) for single_image_tensor in predictions.pred: return_list.append(single_image_tensor) return return_list # using the same label extractor def batch_to_labels(self, batch): formatted_labels = [] for tensor in batch[1]: tensor = torch.index_select(tensor, 1, torch.LongTensor([4, 0, 1, 2, 3])) if len(tensor) > 0 else tensor formatted_labels.append(tensor) return formatted_labels def batch_to_images(self, batch): return batch[0] validate_extractors(CocoDetectionData(data_loader), model) .. rst-class:: sphx-glr-script-out .. code-block:: none Deepchecks will try to validate the extractors given... Structure validation -------------------- Label formatter: Pass! Prediction formatter: Pass! Image formatter: Fail! The data inside the iterable must be a numpy array. Content validation ------------------ For validating the content within the structure you have to manually observe the classes, image, label and prediction. Examples of classes observed in the batch's labels: [[32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 35, 0, 0, 0, 0], [32, 0, 0, 34, 35, 35, 0, 0, 0, 0], [71, 45], [15, 2], [38, 0]] Visual images & label & prediction: Unable to show due to invalid image formatter. .. GENERATED FROM PYTHON SOURCE LINES 134-136 Now our prediction formatter also have valid structure. But in order to really validate it we also need visual assertion and for that we need the image extractor to work. .. GENERATED FROM PYTHON SOURCE LINES 136-166 .. code-block:: default import numpy as np class CocoDetectionData(DetectionData): def batch_to_images(self, batch): # Yolo works on PIL and ImageFormatter expects images as numpy arrays return [np.array(x) for x in batch[0]] # using the same prediction extractor def infer_on_batch(self, batch, model, device): # Convert from yolo Detections object to List (per image) of Tensors of the shape [N, 6]""" return_list = [] predictions = model.to(device)(batch[0]) for single_image_tensor in predictions.pred: return_list.append(single_image_tensor) return return_list # using the same label extractor def batch_to_labels(self, batch): formatted_labels = [] for tensor in batch[1]: tensor = torch.index_select(tensor, 1, torch.LongTensor([4, 0, 1, 2, 3])) if len(tensor) > 0 else tensor formatted_labels.append(tensor) return formatted_labels validate_extractors(CocoDetectionData(data_loader), model) .. rst-class:: sphx-glr-script-out .. code-block:: none Deepchecks will try to validate the extractors given... Structure validation -------------------- Label formatter: Pass! Prediction formatter: Pass! Image formatter: Pass! Content validation ------------------ For validating the content within the structure you have to manually observe the classes, image, label and prediction. Examples of classes observed in the batch's labels: [[32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 35, 0, 0, 0, 0], [32, 0, 0, 34, 35, 35, 0, 0, 0, 0], [71, 45], [15, 2], [38, 0]] Visual images & label & prediction: should open in a new window ******************************************************************************* This machine does not support GUI The formatted image was saved in: /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/vision/tutorials/deepchecks_formatted_image.jpg Visual examples of an image with prediction and label data. Label is red, prediction is blue, and deepchecks loves you. validate_extractors can be set to skip the image saving or change the save path ******************************************************************************* .. GENERATED FROM PYTHON SOURCE LINES 167-172 Now that that image extractor is valid it displays for us visually the label and prediction. When we look at the label we see it is correct, but when we look at the bounding box predictions something seems broken. We need to fix the prediction so the prediction will be returned in [x, y, w, h, confidence, class] format. .. GENERATED FROM PYTHON SOURCE LINES 172-202 .. code-block:: default class CocoDetectionData(DetectionData): def infer_on_batch(self, batch, model, device): # Convert from yolo Detections object to List (per image) of Tensors of the shape [N, 6] with each row being # [x, y, w, h, confidence, class] for each bbox in the image.""" return_list = [] predictions = model.to(device)(batch[0]) # yolo Detections objects have List[torch.Tensor] xyxy output in .pred for single_image_tensor in predictions.pred: pred_modified = torch.clone(single_image_tensor) pred_modified[:, 2] = pred_modified[:, 2] - pred_modified[:, 0] # w = x_right - x_left pred_modified[:, 3] = pred_modified[:, 3] - pred_modified[:, 1] # h = y_bottom - y_top return_list.append(pred_modified) return return_list # using the same label extractor def batch_to_labels(self, batch): formatted_labels = [] for tensor in batch[1]: tensor = torch.index_select(tensor, 1, torch.LongTensor([4, 0, 1, 2, 3])) if len(tensor) > 0 else tensor formatted_labels.append(tensor) return formatted_labels # using the same image extractor def batch_to_images(self, batch): return [np.array(x) for x in batch[0]] .. GENERATED FROM PYTHON SOURCE LINES 203-205 The end result -------------- .. GENERATED FROM PYTHON SOURCE LINES 205-206 .. code-block:: default validate_extractors(CocoDetectionData(data_loader), model) .. rst-class:: sphx-glr-script-out .. code-block:: none Deepchecks will try to validate the extractors given... Structure validation -------------------- Label formatter: Pass! Prediction formatter: Pass! Image formatter: Pass! Content validation ------------------ For validating the content within the structure you have to manually observe the classes, image, label and prediction. Examples of classes observed in the batch's labels: [[32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 35, 0, 0, 0, 0], [32, 0, 0, 34, 35, 35, 0, 0, 0, 0], [71, 45], [15, 2], [38, 0]] Visual images & label & prediction: should open in a new window ******************************************************************************* This machine does not support GUI The formatted image was saved in: /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/vision/tutorials/deepchecks_formatted_image (1).jpg Visual examples of an image with prediction and label data. Label is red, prediction is blue, and deepchecks loves you. validate_extractors can be set to skip the image saving or change the save path ******************************************************************************* .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 1 minutes 38.427 seconds) .. _sphx_glr_download_user-guide_vision_auto_tutorials_plot_extractors_validating.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_extractors_validating.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_extractors_validating.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_