Working with Models and Predictions#

Some checks, mainly the ones related to model evaluation, require model predictions in order to run. In deepchecks, predictions are passed into the suite / check run method in one of the following ways:

Passing a model object that will compute the predictions on the input data.
Passing pre-computed predictions.

Passing pre-computed predictions is a simple alternative to passing a model. It is specifically recommended to use this option if your model object is unavailable locally (for example if placed on a separate prediction server) or if the predicting process is computationally expensive or time consuming.

Supported Tasks and Predictions Format#

Deepchecks currently supports model predictions for regression, binary and multiclass classification tasks. Whether provided from a model interface or as a pre-computed predicted values, the predictions must be in the following format based on the task type:

Predicted values: should be provided as an array-like of shape (n_samples,), containing the predicted value for each sample in the dataset. Predicted values are required for all task types.
Probabilities per class: should be provided as an array-like of shape (n_samples, n_classes) containing the predicted probability of each class per sample. Probabilities per class are only required for classification tasks.

Passing a Model#

Deepchecks requires models to follow the |scikit-learn API conventions| for calculating predicted values and probabilities per class. Therefore built-in scikit-learn classifiers and regressors, along with many additional popular models types (e.g. XGBoost, LightGBM, CatBoost etc.) are supported out of the box.

Specifically, deepchecks requires the following methods to be implemented in the model object:

predict method which receives an array-like of shape (n_samples, n_features) containing the input features and returns predicted values.
predict_proba method which receives an array-like of shape (n_samples, n_features) containing the input features and returns probabilities per class. This method is required only for classification tasks.

Running Deepchecks With a Supported Model#

from deepchecks.tabular.datasets.classification.iris import load_data, load_fitted_model
from deepchecks.tabular.suites import model_evaluation

ds_train, ds_test = load_data(data_format='Dataset', as_train_test=True)
rf_clf = load_fitted_model()  # trained sklearn RandomForestClassifier
result = model_evaluation().run(train_dataset=ds_train, test_dataset=ds_test, model=rf_clf)

Adapting Your Custom Model#

If you are using a model that does not support those interfaces you can either add the required methods to the model’s class or create a wrapper class that implements the required interfaces by calling the relevant APIs of your model. Below is a general structure of such wrapper class.

>>> class MyModelWrapper:
...     def predict(X: pd.DataFrame) -> np.ndarray:
...         # Implement based on base model's API.
...         ...
...     def predict_proba(X: pd.DataFrame) -> np.ndarray:
...         # Implement based on base model's API, only required for classification tasks.
...         ...
...     @property
...     def feature_importances_(self) -> pd.Series:  # optional
...         # Return a pandas Series with feature names as index and their corresponding importance as values.
...         ...

Feature Importance (Optional)#

Some checks uses the model’s feature importance in their analysis. By default, if available, it is extracted directly from the model via property (feature_importances_ or coef_ for a linear model) otherwise it is calculated using sklearn permutation_importance. The required format for the feature importance is a pandas series with feature names as index and their corresponding importance as values.

Using Pre-computed Predictions#

The predictions should be passed via the y_proba and y_pred arguments of the suite / check’s run method in the appropriate format. y_pred receives the predicted values of the model and y_proba receives the probabilities per class, which is only required for classification tasks.

The predictions should be provided for each dataset supplied to the suite / check. For example the Simple Model Comparison check for a regression model requires both train and test predicted values to be provided via the y_pred_train, y_pred_test arguments.

For classification tasks, predicted values are not mandatory. If not supplied, deepchecks will assume the predicted class is the class with the highest predicted probability.

Note

When using pre-computed predictions, if the train dataset shares indices with the test dataset we will add train/test prefixes to the indexes.

Running Deepchecks with Pre-computed Predictions#

We will run the deepchecks model evaluation suite using pre-computed predictions from a random forest classification model. In addition, we will calculate and pass sklearn permutation_importance which provides a better estimate of the effect of different features on the model’s performance. See the feature importance API reference for more details.

from deepchecks.tabular.datasets.classification.iris import load_data, load_fitted_model
from deepchecks.tabular.suites import model_evaluation
from deepchecks.tabular.feature_importance import calculate_feature_importance

ds_train, ds_test = load_data(data_format='Dataset', as_train_test=True)
rf_clf = load_fitted_model()  # trained sklearn RandomForestClassifier

fi = calculate_feature_importance(rf_clf, ds_train)
train_proba = rf_clf.predict_proba(ds_train.features_columns)
test_proba = rf_clf.predict_proba(ds_test.features_columns)

# In classification, predicted values can be supplied via the y_pred_train, y_pred_test
# arguments or inferred from the probabilities per class.
result = model_evaluation().run(train_dataset=ds_train, test_dataset=ds_test,
            features_importance=fi, y_proba_train=train_proba, y_proba_test=test_proba)

The Dataset Object

Feature Importance