.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/model_evaluation/plot_class_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_model_evaluation_plot_class_performance.py: .. _plot_vision_class_performance: Class Performance *********************** This notebooks provides an overview for using and understanding the class performance check. **Structure:** * `What is the purpose of the check? <#what-is-the-purpose-of-the-check>`__ * `Classification <#classification-performance-report>`__ - `Generate data & model <#generate-data-and-model>`__ - `Run the check <#run-the-check>`__ * `Object Detection <#object-detection-class-performance>`__ - `Generate data & model <#id1>`__ - `Run the check <#id2>`__ What Is the Purpose of the Check? ================================= The class performance check evaluates several metrics on the given model and data and returns all of the results in a single check. The check uses the following default metrics: ================= ==================== Task Type Property name ================= ==================== Classification Precision Classification Recall Object Detection `Average Precision `__ Object Detection `Average Recall `__ ================= ==================== In addition to the default metrics, the check supports custom metrics that should be implemented using the `torch.ignite.Metric `__ API. These can be passed as a list using the alternative_metrics parameter of the check, which will override the default metrics. .. GENERATED FROM PYTHON SOURCE LINES 47-49 Imports ------- .. GENERATED FROM PYTHON SOURCE LINES 49-53 .. code-block:: default from deepchecks.vision.checks import ClassPerformance from deepchecks.vision.datasets.classification import mnist .. GENERATED FROM PYTHON SOURCE LINES 54-58 Classification Performance Report ================================= Generate data and model: ------------------------ .. GENERATED FROM PYTHON SOURCE LINES 58-64 .. code-block:: default mnist_model = mnist.load_model() train_ds = mnist.load_dataset(train=True, object_type='VisionData') test_ds = mnist.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 65-67 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 67-72 .. code-block:: default check = ClassPerformance() result = check.run(train_ds, test_ds, mnist_model) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############### | 15/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################## | 30/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################## | 46/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################# | 61/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################# | 77/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################################# | 93/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################################################# | 109/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################################################################# | 125/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################################################################################# | 141/157 [Time: 00:00] Ingesting Batches - Train Dataset: |#############################################################################################################################################################| 157/157 [Time: 00:01] Ingesting Batches - Train Dataset: |#############################################################################################################################################################| 157/157 [Time: 00:01] Ingesting Batches - Test Dataset: | | 0/10 [Time: 00:00] Ingesting Batches - Test Dataset: |## | 2/10 [Time: 00:00] Ingesting Batches - Test Dataset: |#### | 4/10 [Time: 00:00] Ingesting Batches - Test Dataset: |###### | 6/10 [Time: 00:00] Ingesting Batches - Test Dataset: |######## | 8/10 [Time: 00:00] Ingesting Batches - Test Dataset: |##########| 10/10 [Time: 00:00] Ingesting Batches - Test Dataset: |##########| 10/10 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 73-77 If you have a GPU, you can speed up this check by calling: .. code:: python check.run(train_ds, test_ds, mnist_model, device=) .. GENERATED FROM PYTHON SOURCE LINES 79-80 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 80-82 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 83-84 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 86-92 Object Detection Class Performance ================================== For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds. .. GENERATED FROM PYTHON SOURCE LINES 92-95 .. code-block:: default from deepchecks.vision.datasets.detection import coco .. GENERATED FROM PYTHON SOURCE LINES 96-100 Generate Data and Model ----------------------- We generate a sample dataset of 128 images from the `COCO dataset `__, and using the `YOLOv5 model `__. .. GENERATED FROM PYTHON SOURCE LINES 100-106 .. code-block:: default yolo = coco.load_model(pretrained=True) train_ds = coco.load_dataset(train=True, object_type='VisionData') test_ds = coco.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 107-109 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 109-114 .. code-block:: default check = ClassPerformance(show_only='best') result = check.run(train_ds, test_ds, yolo) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:11] Validating Input: |#####| 1/1 [Time: 00:11] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:05] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:11] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:11] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:05] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:11] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:11] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 115-116 If you have a GPU, you can speed up this check by calling: .. GENERATED FROM PYTHON SOURCE LINES 116-119 .. code-block:: default # check.run(train_ds, test_ds, yolo, device=) .. GENERATED FROM PYTHON SOURCE LINES 120-121 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 121-123 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 124-125 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 127-132 Define a Condition ================== We can also define a condition to validate that our model performance is above a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object. .. GENERATED FROM PYTHON SOURCE LINES 132-139 .. code-block:: default check = ClassPerformance(show_only='worst') check.add_condition_test_performance_greater_than(0.2) result = check.run(train_ds, test_ds, yolo) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:11] Validating Input: |#####| 1/1 [Time: 00:11] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:05] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:11] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:11] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:05] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:11] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:11] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 140-141 We detected that for several classes our model performance is below the threshold. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 1 minutes 11.760 seconds) .. _sphx_glr_download_checks_gallery_vision_model_evaluation_plot_class_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_class_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_class_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_