Calibration Metric

.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tabular/auto_checks/model_evaluation/plot_calibration_score.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_tabular_auto_checks_model_evaluation_plot_calibration_score.py: .. _tabular__calibration_score: Calibration Score ******************* This notebook provides an overview for using and understanding the Calibration Score check. **Structure:** * `What is the Calibration Score check? <#what-is-the-calibration-score-check>`__ * `Binary Classification <#binary-classification>`__ * `Multi-class classification <#multi-class-classification>`__ What is the Calibration Score check? ====================================== The ``CalibrationScore`` check calculates the calibration curve with brier score for each class. Calibration refers to the relationship between the model probabilities for one label to the ground truth (the label). For instance, a probability of 0.7 for class A represents that there is 70% chance the true label of this sample is actually class A. Calibration curves (also known as reliability diagrams) compare how well the probabilistic predictions of the classifier are calibrated by plotting the true frequency of one label against its predicted probability. The Brier score metric may be used to assess how well a classifier is calibrated (`Brier score `__). .. GENERATED FROM PYTHON SOURCE LINES 34-36 Imports ======= .. GENERATED FROM PYTHON SOURCE LINES 36-55 .. code-block:: default import warnings import pandas as pd from sklearn.datasets import load_iris from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from deepchecks.tabular import Dataset from deepchecks.tabular.checks import CalibrationScore from deepchecks.tabular.datasets.classification import adult def custom_formatwarning(msg, *args, **kwargs): # ignore everything except the message return str(msg) + '\n' warnings.formatwarning = custom_formatwarning .. GENERATED FROM PYTHON SOURCE LINES 56-58 Binary Classification ======================= .. GENERATED FROM PYTHON SOURCE LINES 58-70 .. code-block:: default # Generate data & model # ----------------------- # The dataset is the adult dataset which can be downloaded from the UCI machine # learning repository. # # Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. # Irvine, CA: University of California, School of Information and Computer Science. train_ds, test_ds = adult.load_data() model = adult.load_fitted_model() .. GENERATED FROM PYTHON SOURCE LINES 71-73 Run the check ---------------- .. GENERATED FROM PYTHON SOURCE LINES 73-77 .. code-block:: default check = CalibrationScore() check.run(test_ds, model) .. raw:: html

Calibration Metric

.. GENERATED FROM PYTHON SOURCE LINES 78-80 Multi-class classification ============================ .. GENERATED FROM PYTHON SOURCE LINES 80-95 .. code-block:: default # Generate data & model # ----------------------- iris = load_iris(as_frame=True) clf = LogisticRegression() frame = iris.frame X = iris.data y = iris.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=55) clf.fit(X_train, y_train) ds = Dataset(pd.concat([X_test, y_test], axis=1), features=iris.feature_names, label='target') .. GENERATED FROM PYTHON SOURCE LINES 96-98 Run the check ---------------- .. GENERATED FROM PYTHON SOURCE LINES 98-100 .. code-block:: default check = CalibrationScore() check.run(ds, clf) .. raw:: html

Calibration Metric

.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 3.887 seconds) .. _sphx_glr_download_tabular_auto_checks_model_evaluation_plot_calibration_score.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_calibration_score.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_calibration_score.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_