.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "vision/auto_checks/model_evaluation/plot_class_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_vision_auto_checks_model_evaluation_plot_class_performance.py: .. _vision__class_performance: Class Performance *********************** This notebooks provides an overview for using and understanding the class performance check. **Structure:** * `What is the purpose of the check? <#what-is-the-purpose-of-the-check>`__ * `Classification <#classification-performance-report>`__ - `Generate Dataset <#generate-dataset>`__ - `Run the check <#run-the-check>`__ * `Object Detection <#object-detection-class-performance>`__ - `Generate data & model <#id1>`__ - `Run the check <#id2>`__ What Is the Purpose of the Check? ================================= The class performance check evaluates several metrics on the given model and data and returns all of the results in a single check. The check uses the following default metrics: ================= ==================== Task Type Property name ================= ==================== Classification Precision Classification Recall Object Detection `Average Precision `__ Object Detection `Average Recall `__ ================= ==================== In addition to the default metrics, the check supports custom metrics, as detailed in the :ref:`Metrics Guide `. These can be passed as a list using the scorers parameter of the check, which will override the default metrics. .. GENERATED FROM PYTHON SOURCE LINES 44-52 Imports ------- .. note:: In this example, we use the pytorch version of the mnist dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.classification import mnist_tensorflow as mnist .. GENERATED FROM PYTHON SOURCE LINES 52-56 .. code-block:: default from deepchecks.vision.checks import ClassPerformance from deepchecks.vision.datasets.classification import mnist_torch as mnist .. GENERATED FROM PYTHON SOURCE LINES 57-61 Classification Performance Report ================================= Generate Dataset ----------------- .. GENERATED FROM PYTHON SOURCE LINES 61-66 .. code-block:: default train_ds = mnist.load_dataset(train=True, object_type='VisionData') test_ds = mnist.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 67-69 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 69-74 .. code-block:: default check = ClassPerformance() result = check.run(train_ds, test_ds) result .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:07] Processing Test Batches: |█████| 1/1 [Time: 00:07] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 75-80 To display the results in an IDE like PyCharm, you can use the following code: .. code-block:: python result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 82-83 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 85-97 Object Detection Class Performance ================================== For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds. .. note:: In this example, we use the pytorch version of the coco dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.detection import coco_tensorflow as coco .. GENERATED FROM PYTHON SOURCE LINES 97-100 .. code-block:: default from deepchecks.vision.datasets.detection import coco_torch as coco .. GENERATED FROM PYTHON SOURCE LINES 101-105 Generate Dataset ---------------- We generate a sample dataset of 128 images from the `COCO dataset `__, and using the `YOLOv5 model `__. .. GENERATED FROM PYTHON SOURCE LINES 105-109 .. code-block:: default train_ds = coco.load_dataset(train=True, object_type='VisionData') test_ds = coco.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 110-112 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 112-117 .. code-block:: default check = ClassPerformance(show_only='best') result = check.run(train_ds, test_ds) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 118-119 If you have a GPU, you can speed up this check by calling: .. GENERATED FROM PYTHON SOURCE LINES 119-122 .. code-block:: default # check.run(train_ds, test_ds, yolo, device=) .. GENERATED FROM PYTHON SOURCE LINES 123-124 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 124-126 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 127-128 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 130-135 Define a Condition ================== We can also define a condition to validate that our model performance is above a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object. .. GENERATED FROM PYTHON SOURCE LINES 135-142 .. code-block:: default check = ClassPerformance(show_only='worst') check.add_condition_test_performance_greater_than(0.2) result = check.run(train_ds, test_ds) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 143-144 We detected that for several classes our model performance is below the threshold. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 16.516 seconds) .. _sphx_glr_download_vision_auto_checks_model_evaluation_plot_class_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_class_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_class_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_