.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "checks_gallery/vision/performance/plot_simple_model_comparison.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_checks_gallery_vision_performance_plot_simple_model_comparison.py: Simple Model Comparison *********************** This notebooks provides an overview for using and understanding simple model comparison check. **Structure:** * `What Is the Purpose of the Check? <#what-is-the-purpose-of-the-check>`__ * `Generate data an model <#generate-data-and-model>`__ * `Run the check <#run-the-check>`__ What Is the Purpose of the Check? ================================= This check compares your current model to a "simple model", which is a model designed to produce the best performance achievable using very simple rules, such as "always predict the most common class". The simple model is used as a **baseline** model; If your model achieves less or similar score to the simple model, this is an indicator of a possible problem with the model (e.g. it wasn't trained properly). Using the parameter ``strategy``, you can select the simple model used in the check: ================ =================================== Strategy Description ================ =================================== prior (default) The probability vector always contains the empirical class prior distribution (i.e. the class distribution observed in the training set). most_frequent The most frequent prediction is predicted. The probability vector is 1 for the most frequent prediction and 0 for the other predictions. stratified The predictions are generated by sampling one-hot vectors from a multinomial distribution parametrized by the empirical class prior probabilities. uniform Generates predictions uniformly at random from the list of unique classes observed in y, i.e. each class has equal probability. ================ =================================== Similiar to the :doc:`tabular simple model comparison check `, there is no simple model which is more "correct" to use, each gives a different baseline to compare to, and you may experiment with the different types and see how it performs on your data. This checks applies only to classification datasets. .. GENERATED FROM PYTHON SOURCE LINES 46-48 Generate data and model ----------------------- .. GENERATED FROM PYTHON SOURCE LINES 48-52 .. code-block:: default from deepchecks.vision.checks.performance import SimpleModelComparison from deepchecks.vision.datasets.classification import mnist .. GENERATED FROM PYTHON SOURCE LINES 53-59 .. code-block:: default mnist_model = mnist.load_model() train_ds = mnist.load_dataset(train=True, object_type='VisionData') test_ds = mnist.load_dataset(train=False, object_type='VisionData') .. GENERATED FROM PYTHON SOURCE LINES 60-65 Run the check ------------- We will run the check with the prior model type. The check will use the default classification metrics - precision and recall. This can be overridden by providing an alternative scorer using the ``alternative_metrics``` parameter. .. GENERATED FROM PYTHON SOURCE LINES 65-69 .. code-block:: default check = SimpleModelComparison(strategy='stratified') result = check.run(train_ds, test_ds, mnist_model) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Validating Input: 0%| | 0/1 [00:00

Simple Model Comparison

Compare given model score to simple model score (according to given model type).

Additional Outputs

Note - data sampling: Running on 10000 train data samples out of 60000. Sample size can be controlled with the "n_samples" parameter.



.. GENERATED FROM PYTHON SOURCE LINES 73-82 Observe the check's output -------------------------- We can see in the results that the check calculates the score for each class in the dataset, and compares the scores between our model and the simple model. In addition to the graphic output, the check also returns a value which includes all of the information that is needed for defining the conditions for validation. The value is a dataframe that contains the metrics' values for each class and dataset: .. GENERATED FROM PYTHON SOURCE LINES 82-85 .. code-block:: default result.value.sort_values(by=['Class', 'Metric']).head(10) .. raw:: html
Model Metric Class Class Name Number of samples Value
5 Simple Model F1 0 0 980 0.095626
10 Perfect Model F1 0 0 980 1.000000
22 Given Model F1 0 0 980 0.986343
0 Simple Model F1 1 1 1135 0.109215
11 Perfect Model F1 1 1 1135 1.000000
20 Given Model F1 1 1 1135 0.992534
4 Simple Model F1 2 2 1032 0.101266
12 Perfect Model F1 2 2 1032 1.000000
24 Given Model F1 2 2 1032 0.985915
3 Simple Model F1 3 3 1010 0.105209


.. GENERATED FROM PYTHON SOURCE LINES 86-103 Define a condition ================== We can define on our check a condition that will validate our model is better than the simple model by a given margin called gain. For classification we check the gain for each class separately and if there is a class that doesn't pass the defined gain the condition will fail. The performance gain is the percent of the improved performance out of the "remaining" unattained performance. Its purpose is to reflect the significance of the said improvement. Take for example for a metric between 0 and 1. A change of only 0.03 that takes us from 0.95 to 0.98 is highly significant (especially in an imbalance scenario), but improving from 0.1 to 0.13 is not a great achievement. The gain is calculated as: :math:`gain = \frac{\text{model score} - \text{simple score}} {\text{perfect score} - \text{simple score}}` Let's add a condition to the check and see what happens when it fails: .. GENERATED FROM PYTHON SOURCE LINES 103-109 .. code-block:: default check = SimpleModelComparison(strategy='stratified') check.add_condition_gain_not_less_than(min_allowed_gain=0.99) result = check.run(train_ds, test_ds, mnist_model) result .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Validating Input: 0%| | 0/1 [00:00

Simple Model Comparison

Compare given model score to simple model score (according to given model type).

Conditions Summary
Status Condition More Info
✖
 Model performance gain over simple model is not less than 99% Found metrics with gain below threshold: {'F1': {6: '98.79%', 0: '98.49%', 3: '98.47%', 2: '98.45%', 4: '98.43%', 5: '98.03%', 8: '97.88%', 7: '97.65%', 9: '96.91%'}}
Additional Outputs

Note - data sampling: Running on 10000 train data samples out of 60000. Sample size can be controlled with the "n_samples" parameter.



.. GENERATED FROM PYTHON SOURCE LINES 110-112 We detected that for several classes our gain did not passed the target gain we defined, therefore it failed. .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 3.382 seconds) .. _sphx_glr_download_checks_gallery_vision_performance_plot_simple_model_comparison.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_simple_model_comparison.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_simple_model_comparison.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_