.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/model_evaluation/plot_model_error_analysis.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_model_evaluation_plot_model_error_analysis.py: .. _plot_vision_model_error_analysis: Model Error Analysis check ========================== This notebooks provides an overview for using and understanding the model error analysis check. **Structure:** - `What is the purpose of the check? <#what-is-the-purpose-of-the-check>`__ - `Classification <#classification-performance-report>`__ - `Generate data & model <#generate_c>`__ - `Run the check <#run_check_c>`__ - `Object Detection <#object-detection-class-performance>`__ - `Generate data & model <#generate_o>`__ - `Run the check <#run_check_o>`__ What is the purpose of the check? --------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 30-32 Imports ------- .. GENERATED FROM PYTHON SOURCE LINES 32-34 .. code-block:: default from deepchecks.vision.checks import ModelErrorAnalysis .. GENERATED FROM PYTHON SOURCE LINES 35-42 Classification Performance Report --------------------------------- .. _generate_c: Generate data and model: ~~~~~~~~~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 42-48 .. code-block:: default from deepchecks.vision.datasets.classification import mnist mnist_model = mnist.load_model() train_ds = mnist.load_dataset(train=True, object_type='VisionData') test_ds = mnist.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 49-53 .. _run_check_c: Run the check: ~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 53-57 .. code-block:: default check = ModelErrorAnalysis(min_error_model_score=-0.1) result = check.run(train_ds, test_ds, mnist_model) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:00] Ingesting Batches - Train Dataset: | | 0/157 [Time: 00:00] Ingesting Batches - Train Dataset: |########### | 11/157 [Time: 00:00] Ingesting Batches - Train Dataset: |###################### | 22/157 [Time: 00:00] Ingesting Batches - Train Dataset: |################################# | 33/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################ | 44/157 [Time: 00:00] Ingesting Batches - Train Dataset: |####################################################### | 55/157 [Time: 00:00] Ingesting Batches - Train Dataset: |################################################################## | 66/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################# | 77/157 [Time: 00:00] Ingesting Batches - Train Dataset: |######################################################################################## | 88/157 [Time: 00:00] Ingesting Batches - Train Dataset: |################################################################################################### | 99/157 [Time: 00:00] Ingesting Batches - Train Dataset: |############################################################################################################## | 110/157 [Time: 00:01] Ingesting Batches - Train Dataset: |######################################################################################################################### | 121/157 [Time: 00:01] Ingesting Batches - Train Dataset: |#################################################################################################################################### | 132/157 [Time: 00:01] Ingesting Batches - Train Dataset: |############################################################################################################################################### | 143/157 [Time: 00:01] Ingesting Batches - Train Dataset: |########################################################################################################################################################## | 154/157 [Time: 00:01] Ingesting Batches - Train Dataset: |#############################################################################################################################################################| 157/157 [Time: 00:01] Ingesting Batches - Test Dataset: | | 0/10 [Time: 00:00] Ingesting Batches - Test Dataset: |# | 1/10 [Time: 00:00] Ingesting Batches - Test Dataset: |## | 2/10 [Time: 00:00] Ingesting Batches - Test Dataset: |### | 3/10 [Time: 00:00] Ingesting Batches - Test Dataset: |#### | 4/10 [Time: 00:00] Ingesting Batches - Test Dataset: |##### | 5/10 [Time: 00:00] Ingesting Batches - Test Dataset: |###### | 6/10 [Time: 00:00] Ingesting Batches - Test Dataset: |####### | 7/10 [Time: 00:00] Ingesting Batches - Test Dataset: |######## | 8/10 [Time: 00:01] Ingesting Batches - Test Dataset: |######### | 9/10 [Time: 00:01] Ingesting Batches - Test Dataset: |##########| 10/10 [Time: 00:01] Ingesting Batches - Test Dataset: |##########| 10/10 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:122: FutureWarning: Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327 /home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:127: FutureWarning: Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327 Computing Check: |#####| 1/1 [Time: 00:04] Computing Check: |#####| 1/1 [Time: 00:04] .. raw:: html
Model Error Analysis


.. GENERATED FROM PYTHON SOURCE LINES 58-61 If you have a GPU, you can speed up this check by passing it as an argument to .run() as device= To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 61-63 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 64-65 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 68-75 Object Detection Class Performance ---------------------------------- For object detection tasks - the default metric that is being calculated it the Average Precision. The definition of the Average Precision is identical to how the COCO dataset defined it - mean of the average precision per class, over the range [0.5, 0.95, 0.05] of IoU thresholds. .. GENERATED FROM PYTHON SOURCE LINES 75-80 .. code-block:: default import numpy as np from deepchecks.vision.datasets.detection import coco .. GENERATED FROM PYTHON SOURCE LINES 81-89 .. _generate_o: Generate Data and Model ~~~~~~~~~~~~~~~~~~~~~~~ We generate a sample dataset of 128 images from the `COCO dataset `__, and using the `YOLOv5 model `__ .. GENERATED FROM PYTHON SOURCE LINES 89-95 .. code-block:: default yolo = coco.load_model(pretrained=True) train_ds = coco.load_dataset(train=True, object_type='VisionData') test_ds = coco.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 96-100 .. _run_check_o: Run the check: ~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 100-104 .. code-block:: default check = ModelErrorAnalysis(min_error_model_score=-1) result = check.run(train_ds, test_ds, yolo) result .. rst-class:: sphx-glr-script-out .. code-block:: none Validating Input: | | 0/1 [Time: 00:00] Validating Input: |#####| 1/1 [Time: 00:09] Validating Input: |#####| 1/1 [Time: 00:09] Ingesting Batches - Train Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Train Dataset: |##5 | 1/2 [Time: 00:06] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:12] Ingesting Batches - Train Dataset: |#####| 2/2 [Time: 00:12] Ingesting Batches - Test Dataset: | | 0/2 [Time: 00:00] Ingesting Batches - Test Dataset: |##5 | 1/2 [Time: 00:05] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:12] Ingesting Batches - Test Dataset: |#####| 2/2 [Time: 00:12] Computing Check: | | 0/1 [Time: 00:00]/home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:122: FutureWarning: Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327 /home/runner/work/deepchecks/deepchecks/venv/lib/python3.9/site-packages/category_encoders/target_encoder.py:127: FutureWarning: Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327 Computing Check: |#####| 1/1 [Time: 00:02] Computing Check: |#####| 1/1 [Time: 00:02] .. raw:: html
Model Error Analysis


.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 44.364 seconds) .. _sphx_glr_download_checks_gallery_vision_model_evaluation_plot_model_error_analysis.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_model_error_analysis.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_model_error_analysis.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_