Image Segment Performance

.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/model_evaluation/plot_image_segment_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_model_evaluation_plot_image_segment_performance.py: .. _plot_vision_image_segment_performance: Image Segment Performance ************************* This notebooks provides an overview for using and understanding image segment performance check. **Structure:** * `Why the image segment performance is important? <#why-segment-performance-is-important>`__ * `Run the check <#run-the-check>`__ * `Define a condition <#define-a-condition>`__ Why segment performance is important? ===================================== The check helps to detect segments of your data that are under-performing based on the basic properties of the image. For example, by default the check would show how the performance depends on brightness, area and other such properties. Identifying your models' weak segments might help to address specific issues and improve the overall performance of the model. .. GENERATED FROM PYTHON SOURCE LINES 26-28 Run the check ============= .. GENERATED FROM PYTHON SOURCE LINES 28-38 .. code-block:: default from deepchecks.vision.checks import ImageSegmentPerformance from deepchecks.vision.datasets.detection import coco coco_data = coco.load_dataset(train=False, object_type='VisionData') model = coco.load_model() result = ImageSegmentPerformance().run(coco_data, model) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:04] Validating Input: |#####| 1/1 [Time: 00:04] Ingesting Batches: | | 0/2 [Time: 00:00] Ingesting Batches: |##5 | 1/2 [Time: 00:05] Ingesting Batches: |#####| 2/2 [Time: 00:12] Ingesting Batches: |#####| 2/2 [Time: 00:12] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:02] Computing Check: |#####| 1/1 [Time: 00:02] .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 39-42 If you have a GPU, you can speed up this check by passing it as an argument to .run() as device= To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 42-44 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 45-46 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 49-53 Observe the check’s output -------------------------- The check segmented the data by different properties and calculated the metrics for each segment. As the value of result we return all the information on the different segments: .. GENERATED FROM PYTHON SOURCE LINES 53-57 .. code-block:: default print(f'Properties: {result.value.keys()}') print(f'brightness bins: {result.value["Brightness"]}') .. rst-class:: sphx-glr-script-out .. code-block:: none Properties: dict_keys(['Aspect Ratio', 'Area', 'Brightness', 'RMS Contrast', 'Mean Red Relative Intensity', 'Mean Green Relative Intensity', 'Mean Blue Relative Intensity']) brightness bins: [{'start': -inf, 'stop': 0.3283433811152742, 'count': 13, 'metrics': {'Average Precision': 0.4544554455445543, 'Average Recall': 0.4666666666666667}, 'display_range': '(-inf, 0.33)'}, {'start': 0.3283433811152742, 'stop': 0.4030219196204722, 'count': 13, 'metrics': {'Average Precision': 0.3029702970297029, 'Average Recall': 0.3}, 'display_range': '[0.33, 0.4)'}, {'start': 0.4030219196204722, 'stop': 0.4620466695414141, 'count': 12, 'metrics': {'Average Precision': 0.20495049504950488, 'Average Recall': 0.2}, 'display_range': '[0.4, 0.46)'}, {'start': 0.4620466695414141, 'stop': 0.5060984084251257, 'count': 13, 'metrics': {'Average Precision': 0.4544554455445543, 'Average Recall': 0.45}, 'display_range': '[0.46, 0.51)'}, {'start': 0.5060984084251257, 'stop': inf, 'count': 13, 'metrics': {'Average Precision': 0.2524752475247524, 'Average Recall': 0.25}, 'display_range': '[0.51, inf)'}] .. GENERATED FROM PYTHON SOURCE LINES 58-67 Define a condition ------------------ The check has a default condition which can be defined. The condition calculates for each property & metric the mean score and then looks at the ratio between the lowest segment score and the mean score. If this ratio is less than defined threshold, the condition fails. The purpose of the condition is to catch properties segments that are significantly worse than the mean - which might indicate a problem. .. GENERATED FROM PYTHON SOURCE LINES 67-72 .. code-block:: default check = ImageSegmentPerformance().add_condition_score_from_mean_ratio_greater_than(0.5) result = check.run(coco_data, model) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:04] Validating Input: |#####| 1/1 [Time: 00:04] Ingesting Batches: | | 0/2 [Time: 00:00] Ingesting Batches: |##5 | 1/2 [Time: 00:05] Ingesting Batches: |#####| 2/2 [Time: 00:12] Ingesting Batches: |#####| 2/2 [Time: 00:12] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:02] Computing Check: |#####| 1/1 [Time: 00:02] .. raw:: html

Image Segment Performance

.. GENERATED FROM PYTHON SOURCE LINES 73-75 In this case the condition identified under-performing segments in the properties: mean_blue_relative_intensity, brightness, aspect_ratio, mean_red_relative_intensity .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 39.596 seconds) .. _sphx_glr_download_checks_gallery_vision_model_evaluation_plot_image_segment_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_image_segment_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_image_segment_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_