.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "vision/auto_checks/model_evaluation/plot_class_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_vision_auto_checks_model_evaluation_plot_class_performance.py: .. _vision__class_performance: Class Performance *********************** This notebooks provides an overview for using and understanding the class performance check. **Structure:** * `What is the purpose of the check? <#what-is-the-purpose-of-the-check>`__ * `Classification <#classification-performance-report>`__ - `Generate Dataset <#generate-dataset>`__ - `Run the check <#run-the-check>`__ * `Object Detection <#object-detection-class-performance>`__ - `Generate data & model <#id1>`__ - `Run the check <#id2>`__ What Is the Purpose of the Check? ================================= The class performance check evaluates several metrics on the given model and data and returns all of the results in a single check. The check uses the following default metrics: ================= ==================== Task Type Property name ================= ==================== Classification Precision Classification Recall Object Detection `Average Precision `__ Object Detection `Average Recall `__ ================= ==================== In addition to the default metrics, the check supports custom metrics, as detailed in the :ref:`Metrics Guide `. These can be passed as a list using the scorers parameter of the check, which will override the default metrics. .. GENERATED FROM PYTHON SOURCE LINES 44-52 Imports ------- .. note:: In this example, we use the pytorch version of the mnist dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.classification import mnist_tensorflow as mnist .. GENERATED FROM PYTHON SOURCE LINES 52-56 .. code-block:: default from deepchecks.vision.checks import ClassPerformance from deepchecks.vision.datasets.classification import mnist_torch as mnist .. GENERATED FROM PYTHON SOURCE LINES 57-61 Classification Performance Report ================================= Generate Dataset ----------------- .. GENERATED FROM PYTHON SOURCE LINES 61-66 .. code-block:: default train_ds = mnist.load_dataset(train=True, object_type='VisionData') test_ds = mnist.load_dataset(train=False, object_type='VisionData') .. rst-class:: sphx-glr-script-out .. code-block:: none You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. .. GENERATED FROM PYTHON SOURCE LINES 67-69 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 69-74 .. code-block:: default check = ClassPerformance() result = check.run(train_ds, test_ds) result .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Train Batches: |█████| 1/1 [Time: 00:01] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:05] Processing Test Batches: |█████| 1/1 [Time: 00:05] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 75-80 To display the results in an IDE like PyCharm, you can use the following code: .. code-block:: python result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 82-83 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 85-97 Object Detection Class Performance ================================== For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds. .. note:: In this example, we use the pytorch version of the coco dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.detection import coco_tensorflow as coco .. GENERATED FROM PYTHON SOURCE LINES 97-100 .. code-block:: default from deepchecks.vision.datasets.detection import coco_torch as coco .. GENERATED FROM PYTHON SOURCE LINES 101-105 Generate Dataset ---------------- We generate a sample dataset of 128 images from the `COCO dataset `__, and using the `YOLOv5 model `__. .. GENERATED FROM PYTHON SOURCE LINES 105-109 .. code-block:: default train_ds = coco.load_dataset(train=True, object_type='VisionData') test_ds = coco.load_dataset(train=False, object_type='VisionData') .. rst-class:: sphx-glr-script-out .. code-block:: none You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. .. GENERATED FROM PYTHON SOURCE LINES 110-112 Run the check ------------- .. GENERATED FROM PYTHON SOURCE LINES 112-117 .. code-block:: default check = ClassPerformance(show_only='best') result = check.run(train_ds, test_ds) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 118-119 If you have a GPU, you can speed up this check by calling: .. GENERATED FROM PYTHON SOURCE LINES 119-122 .. code-block:: default # check.run(train_ds, test_ds, yolo, device=) .. GENERATED FROM PYTHON SOURCE LINES 123-124 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 124-126 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 127-128 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 130-135 Define a Condition ================== We can also define a condition to validate that our model performance is above a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object. .. GENERATED FROM PYTHON SOURCE LINES 135-142 .. code-block:: default check = ClassPerformance(show_only='worst') check.add_condition_test_performance_greater_than(0.2) result = check.run(train_ds, test_ds) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Train Batches: | | 0/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Train Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: | | 0/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Processing Test Batches: |█████| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html
Class Performance


.. GENERATED FROM PYTHON SOURCE LINES 143-144 We detected that for several classes our model performance is below the threshold. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 11.444 seconds) .. _sphx_glr_download_vision_auto_checks_model_evaluation_plot_class_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_class_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_class_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_