.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/model_evaluation/plot_single_dataset_scalar_performance.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_model_evaluation_plot_single_dataset_scalar_performance.py: Single Dataset Scalar Performance ********************************* This notebooks provides an overview for using and understanding single dataset scalar performance check. **Structure:** * `What Is the Purpose of the Check? <#what-is-the-purpose-of-the-check>`__ * `Generate data an model <#generate-data-and-model>`__ * `Run the check <#run-the-check>`__ * `Define a condition <#define-a-condition>`__ What Is the Purpose of the Check? ================================= This check returns a metric result as a single scalar, which is especially useful for monitoring a model in production. Some metrics return a single score, but others return a tensor of scores. For example, Precision returns a tensor in the size of the number of classes. In that case, we will use a reduce function - a function that aggregates the scores into a scalar. In this example we use 'nanmean' that returns the mean over the classes, while ignoring NaNs. .. GENERATED FROM PYTHON SOURCE LINES 27-29 Generate data and model ----------------------- .. GENERATED FROM PYTHON SOURCE LINES 29-33 .. code-block:: default from deepchecks.vision.checks import SingleDatasetScalarPerformance from deepchecks.vision.datasets.classification import mnist .. GENERATED FROM PYTHON SOURCE LINES 34-39 .. code-block:: default mnist_model = mnist.load_model() train_ds = mnist.load_dataset(train=True, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 40-47 Run the check ------------- We will run the check with the model defined above. The check will use the default classification metric - `ignite.Accuracy `__. The default metric returns a scalar, therefore we will use the reduce function default - None. .. GENERATED FROM PYTHON SOURCE LINES 47-53 .. code-block:: default check = SingleDatasetScalarPerformance() result = check.run(train_ds, mnist_model) result .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Validating Input: | | 0/1 [00:00 Single Dataset Scalar Performance

.. GENERATED FROM PYTHON SOURCE LINES 54-58 The result value is a dictionary with the following fields: score - the actual result, metric - the name of metric used reduce - the name of the reduce function used. .. GENERATED FROM PYTHON SOURCE LINES 58-60 .. code-block:: default result.value .. rst-class:: sphx-glr-script-out Out: .. code-block:: none {'score': 0.9837, 'metric': 'accuracy', 'reduce': None} .. GENERATED FROM PYTHON SOURCE LINES 61-63 Now we will run a check with parameters, to use a metric and a reduce function different from the defaults. We will also pass names for them, so that the return value will look neat. .. GENERATED FROM PYTHON SOURCE LINES 63-70 .. code-block:: default from ignite.metrics import Precision from torch import nanmean check = SingleDatasetScalarPerformance(Precision(), nanmean, metric_name='precision', reduce_name='mean') result = check.run(train_ds, mnist_model) result.value .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Validating Input: | | 0/1 [00:00 Single Dataset Scalar Performance

.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 5.474 seconds) .. _sphx_glr_download_checks_gallery_vision_model_evaluation_plot_single_dataset_scalar_performance.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_single_dataset_scalar_performance.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_single_dataset_scalar_performance.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_