.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/model_evaluation/plot_weak_segments_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_model_evaluation_plot_weak_segments_performance.py: Weak Segments Performance ************************* This notebook provides an overview for using and understanding the weak segment performance check. **Structure:** * `What is the purpose of the check? <#what-is-the-purpose-of-the-check>`__ * `Automatically detecting weak segments <#automatically-detecting-weak-segments>`__ * `Generate Dataset <#generate-dataset>`__ * `Run the check <#run-the-check>`__ * `Define a condition <#define-a-condition>`__ What is the purpose of the check? ================================== The check is designed to easily identify the model's weakest segments. The segments are characterized by the :doc:`image properties ` such as contrast and aspect ratio. Automatically detecting weak segments ===================================== The check performs several steps: #. We calculate the image properties for each sample. The properties to calculate can be passed explicitly or resort to the default image properties. #. We calculate loss for each sample in the dataset using the provided model or predictions, the loss function can be passed explicitly or set to a default based on the task type. #. We train multiple simple tree based models, each one is trained using two properties to predict the per sample error calculated before. #. We convert each of the leafs in each of the trees into a segment and calculate the segment's performance. For the weakest segments detected we also calculate the model's performance on the data segments surrounding them. .. GENERATED FROM PYTHON SOURCE LINES 42-50 Generate Dataset ================= .. note:: In this example, we use the pytorch version of the coco dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.detection import coco_tensorflow as coco .. GENERATED FROM PYTHON SOURCE LINES 50-56 .. code-block:: default from deepchecks.vision.checks import WeakSegmentsPerformance from deepchecks.vision.datasets.detection import coco_torch as coco coco_data = coco.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 57-59 Run the check ============= .. GENERATED FROM PYTHON SOURCE LINES 59-63 .. code-block:: default check = WeakSegmentsPerformance() result = check.run(coco_data) result .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |#####| 1/1 [Time: 00:00] Processing Batches: |#####| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:04] Computing Check: |#####| 1/1 [Time: 00:04] .. raw:: html
Weak Segments Performance


.. GENERATED FROM PYTHON SOURCE LINES 64-65 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 65-67 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 68-77 The result will be displayed in a new window. %% Observe the check's output -------------------------- We see in the results that the check indeed found several segments on which the model performance is below average. In the heatmap display we can see the model's performance on the weakest segments and their environment with respect to the two segmentation features. In order to get the full list of weak segments found we can look at the result.value attribute. .. GENERATED FROM PYTHON SOURCE LINES 77-81 .. code-block:: default result.value['weak_segments_list'] .. raw:: html
mean IoU score Feature1 Feature1 range Feature2 Feature2 range % of data
2 0.551099 Aspect Ratio (0.6875679194927216, 1.2355396747589111) RMS Contrast (55.4311408996582, 69.72321319580078) 15.6250
1 0.579780 Aspect Ratio (0.6875679194927216, 1.2355396747589111) Brightness (114.77898406982422, inf) 20.3125
5 0.596931 RMS Contrast (62.37001991271973, 69.63874816894531) None 18.7500
7 0.603122 Brightness (89.9817886352539, inf) RMS Contrast (62.00737762451172, 69.63874816894531) 15.6250
9 0.608905 RMS Contrast (-inf, 69.78135681152344) Mean Red Relative Intensity (-inf, 0.3267597407102585) 15.6250
8 0.632636 Mean Red Relative Intensity (-inf, 0.31693054735660553) None 15.6250
3 0.642269 Mean Red Relative Intensity (-inf, 0.31777942180633545) None 17.1875
0 0.659983 Aspect Ratio (0.6875679194927216, 1.2355396747589111) None 48.4375
4 0.670180 Brightness (89.9817886352539, 115.23897171020508) None 28.1250


.. GENERATED FROM PYTHON SOURCE LINES 82-84 Now we will run a check with properties and minimum segment size ratio (the minimal fraction of the data to be considered as a segment) different from the defaults. .. GENERATED FROM PYTHON SOURCE LINES 84-92 .. code-block:: default from deepchecks.vision.utils.image_properties import brightness, texture_level properties = [{'name': 'brightness', 'method': brightness, 'output_type': 'numerical'}, {'name': ' texture', 'method': texture_level, 'output_type': 'numerical'}] check = WeakSegmentsPerformance(segment_minimum_size_ratio=0.03, image_properties=properties) result = check.run(coco_data) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |#####| 1/1 [Time: 00:00] Processing Batches: |#####| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Weak Segments Performance


.. GENERATED FROM PYTHON SOURCE LINES 93-98 Define a condition ================== We can define on our check a condition that will validate that the ratio of the model performance on the weakest segment to the average model performance is less than a specified ratio. .. GENERATED FROM PYTHON SOURCE LINES 98-104 .. code-block:: default # Let's add a condition and re-run the check: check.add_condition_segments_relative_performance_greater_than(0.1) result = check.run(coco_data) result.show(show_additional_outputs=False) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |#####| 1/1 [Time: 00:00] Processing Batches: |#####| 1/1 [Time: 00:00] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] Computing Check: |#####| 1/1 [Time: 00:00] .. raw:: html
Weak Segments Performance


.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 7.785 seconds) .. _sphx_glr_download_checks_gallery_vision_model_evaluation_plot_weak_segments_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_weak_segments_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_weak_segments_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_