.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/performance/plot_model_error_analysis.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_performance_plot_model_error_analysis.py: Model Error Analysis check ========================== This notebooks provides an overview for using and understanding the model error analysis check. **Structure:** - `What is the purpose of the check? <#what-is-the-purpose-of-the-check>`__ - `Classification <#classification-performance-report>`__ - `Generate data & model <#generate_c>`__ - `Run the check <#run_check_c>`__ - `Object Detection <#object-detection-class-performance>`__ - `Generate data & model <#generate_o>`__ - `Run the check <#run_check_o>`__ What is the purpose of the check? --------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 28-30 Imports ------- .. GENERATED FROM PYTHON SOURCE LINES 30-32 .. code-block:: default from deepchecks.vision.checks.performance import ModelErrorAnalysis .. GENERATED FROM PYTHON SOURCE LINES 33-40 Classification Performance Report --------------------------------- .. _generate_c: Generate data and model: ~~~~~~~~~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 40-46 .. code-block:: default from deepchecks.vision.datasets.classification import mnist mnist_model = mnist.load_model() train_ds = mnist.load_dataset(train=True, object_type='VisionData') test_ds = mnist.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 47-51 .. _run_check_c: Run the check: ~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 51-54 .. code-block:: default check = ModelErrorAnalysis(min_error_model_score=-0.1) check.run(train_ds, test_ds, mnist_model) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Validating Input: 0%| | 0/1 [00:00

Model Error Analysis

Find the properties that best split the data into segments of high and low model error.

Additional Outputs
The following graphs show the distribution of error for top properties that are most useful for distinguishing high error samples from low error samples.

Note - data sampling: Running on 10000 train data samples out of 60000. Sample size can be controlled with the "n_samples" parameter.



.. GENERATED FROM PYTHON SOURCE LINES 55-62 Object Detection Class Performance ---------------------------------- For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds. .. GENERATED FROM PYTHON SOURCE LINES 62-67 .. code-block:: default import numpy as np from deepchecks.vision.datasets.detection import coco .. GENERATED FROM PYTHON SOURCE LINES 68-76 .. _generate_o: Generate Data and Model ~~~~~~~~~~~~~~~~~~~~~~~ We generate a sample dataset of 128 images from the `COCO dataset `__, and using the `YOLOv5 model `__ .. GENERATED FROM PYTHON SOURCE LINES 76-82 .. code-block:: default yolo = coco.load_model(pretrained=True) train_ds = coco.load_dataset(train=True, object_type='VisionData') test_ds = coco.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 83-87 .. _run_check_o: Run the check: ~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 87-90 .. code-block:: default check = ModelErrorAnalysis(min_error_model_score=-1) check.run(train_ds, test_ds, yolo) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Validating Input: 0%| | 0/1 [00:00

Model Error Analysis

Find the properties that best split the data into segments of high and low model error.

Additional Outputs
The following graphs show the distribution of error for top properties that are most useful for distinguishing high error samples from low error samples.


.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 46.869 seconds) .. _sphx_glr_download_checks_gallery_vision_performance_plot_model_error_analysis.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_model_error_analysis.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_model_error_analysis.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_