Single Dataset Performance

.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "vision/auto_checks/model_evaluation/plot_single_dataset_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_vision_auto_checks_model_evaluation_plot_single_dataset_performance.py: .. _vision__single_dataset_performance: Single Dataset Performance ********************************* This notebooks provides an overview for using and understanding single dataset performance check. **Structure:** * `What Is the Purpose of the Check? <#what-is-the-purpose-of-the-check>`__ * `Generate Dataset <#generate-dataset>`__ * `Run the check <#run-the-check>`__ * `Define a condition <#define-a-condition>`__ What Is the Purpose of the Check? ================================= This check returns the results from a dict of metrics, in the format metric name: scorer, calculated for the given model dataset. The scorer should be either a sklearn scorer or a custom metric (see :ref:`metrics_user_guide` for further details). Use this check to evaluate the performance on a single vision dataset such as a test set. .. GENERATED FROM PYTHON SOURCE LINES 26-34 Generate Dataset ---------------- .. note:: In this example, we use the pytorch version of the mnist dataset and model. In order to run this example using tensorflow, please change the import statements to:: from deepchecks.vision.datasets.classification import mnist_tensorflow as mnist .. GENERATED FROM PYTHON SOURCE LINES 34-38 .. code-block:: default from deepchecks.vision.checks import SingleDatasetPerformance from deepchecks.vision.datasets.classification import mnist_torch as mnist .. GENERATED FROM PYTHON SOURCE LINES 39-42 .. code-block:: default train_ds = mnist.load_dataset(train=True, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 43-47 Run the check ------------- The check will use the default classification metrics - precision and recall. .. GENERATED FROM PYTHON SOURCE LINES 47-53 .. code-block:: default check = SingleDatasetPerformance() result = check.run(train_ds) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |█████| 1/1 [Time: 00:01] Processing Batches: |█████| 1/1 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 54-55 To display the results in an IDE like PyCharm, you can use the following code: .. GENERATED FROM PYTHON SOURCE LINES 55-57 .. code-block:: default # result.show_in_window() .. GENERATED FROM PYTHON SOURCE LINES 58-59 The result will be displayed in a new window. .. GENERATED FROM PYTHON SOURCE LINES 61-63 Now we will run a check with a metric different from the defaults- F-1. You can read more about setting metrics in the :ref:`Metrics Guide `. .. GENERATED FROM PYTHON SOURCE LINES 63-68 .. code-block:: default check = SingleDatasetPerformance(scorers={'f1': 'f1_per_class'}) result = check.run(train_ds) result .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |█████| 1/1 [Time: 00:01] Processing Batches: |█████| 1/1 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html

Single Dataset Performance

.. GENERATED FROM PYTHON SOURCE LINES 69-74 Define a Condition ================== We can define a condition to validate that our model performance score is above or below a certain threshold. The condition is defined as a function that takes the results of the check as input and returns a ConditionResult object. .. GENERATED FROM PYTHON SOURCE LINES 74-80 .. code-block:: default check = SingleDatasetPerformance() check.add_condition_greater_than(0.5) result = check.run(train_ds) result.show(show_additional_outputs=False) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |█████| 1/1 [Time: 00:02] Processing Batches: |█████| 1/1 [Time: 00:02] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html

Single Dataset Performance

.. GENERATED FROM PYTHON SOURCE LINES 81-83 We can also define a condition on a specific metric (or a subset of the metrics) that was passed to the check and a specific class, instead of testing all the metrics and all the classes which is the default mode. .. GENERATED FROM PYTHON SOURCE LINES 83-88 .. code-block:: default check = SingleDatasetPerformance() check.add_condition_greater_than(0.8, metrics=['Precision'], class_mode='3') result = check.run(train_ds) result.show(show_additional_outputs=False) .. rst-class:: sphx-glr-script-out .. code-block:: none Processing Batches: | | 0/1 [Time: 00:00] Processing Batches: |█████| 1/1 [Time: 00:01] Processing Batches: |█████| 1/1 [Time: 00:01] Computing Check: | | 0/1 [Time: 00:00] Computing Check: |█████| 1/1 [Time: 00:00] .. raw:: html

Single Dataset Performance

.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 8.507 seconds) .. _sphx_glr_download_vision_auto_checks_model_evaluation_plot_single_dataset_performance.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_single_dataset_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_single_dataset_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_