Binder badge Colab badge

Quickstart in 5 minutes

In order to run your first Deepchecks Suite all you need to have is the data and model that you wish to validate. More specifically, you need:

  • DataFrames with your train and test data

  • (optional) A model that supports the scikit-learn API. Required for running checks that need the model’s predictions for running.

To run your first suite you need only a few lines of code, that start here Define a Dataset Object.

# If you don't have deepchecks installed yet:
import sys
!{sys.executable} -m pip install deepchecks --quiet #--user

Load Data, Split Train-Val, and Train a Simple Model

For the purpose of this guide we’ll use the simple iris dataset and train a simple random forest model for multiclass classification:

# General imports
import pandas as pd
import numpy as np

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Load Data
iris_df = load_iris(return_X_y=False, as_frame=True)['frame']
label_col = 'target'
df_train, df_test = train_test_split(iris_df, stratify=iris_df[label_col], random_state=0)

# Train Model
rf_clf = RandomForestClassifier(), axis=1), df_train[label_col]);

Define a Dataset Object

Initialize the Dataset object, stating the relevant metadata about the dataset (e.g. the name for the label column) Check out the Dataset’s attributes to see which additional special columns can be declared and used (e.g. date column, index column).

from deepchecks import Dataset

# We explicitly state that this dataset has no categorical features, otherwise they will be automatically inferred
# If the dataset has categorical features, the best practice is to pass a list with their names

ds_train = Dataset(df_train, label=label_col, cat_features=[])
ds_test =  Dataset(df_test,  label=label_col, cat_features=[])

Run the Full Suite

Use the full_suite that is a collection of (most of) the prebuilt checks. Check out the list of all of the prebuilt suites for some more info about the existing suites.

from deepchecks.suites import full_suite

suite = full_suite()
# note that we set check_datasets_policy to 'both' so that for single dataset checks (e.g. some of the integrity checks),
# both the test and the train data will be checked., test_dataset=ds_test, model=rf_clf)

Inspect suite and remove a condition

We can see that the single feature contribution failed, both for test and for train. Since this is a very simple dataset with few features and this behavior is not necessarily problematic, we will remove the existing conditions for the PPS

# Lets first print the suite to find the conditions that we want to change:
Full Suite: [
        0: ModelInfo
        1: ColumnsInfo
        2: ConfusionMatrixReport
        3: PerformanceReport
                        0: Train-Test scores relative degradation is not greater than 0.1
        4: RocReport(excluded_classes=[])
                        0: AUC score for all the classes is not less than 0.7
        5: SimpleModelComparison
                        0: Model performance gain over simple model is not less than 10%
        6: ModelErrorAnalysis
                        0: The performance difference of the detected segments must not be greater than 5%
        7: CalibrationScore
        8: TrustScoreComparison
                        0: Mean trust score decline is not greater than 20%
        9: RegressionSystematicError
                        0: Bias ratio is not greater than 0.01
        10: RegressionErrorDistribution
                        0: Kurtosis value is not less than -0.1
        11: BoostingOverfit
                        0: Test score over iterations doesn't decline by more than 5% from the best score
        12: UnusedFeatures
                        0: Number of high variance unused features is not greater than 5
        13: ModelInferenceTime
                        0: Average model inference time for one sample is not greater than 0.001
        14: TrainTestFeatureDrift
                        0: PSI <= 0.2 and Earth Mover's Distance <= 0.1
        15: TrainTestLabelDrift
                        0: PSI <= 0.2 and Earth Mover's Distance <= 0.1 for label drift
        16: WholeDatasetDrift
                        0: Drift value is not greater than 0.25
        17: DominantFrequencyChange
                        0: Change in ratio of dominant value in data is not greater than 25%
        18: CategoryMismatchTrainTest
                        0: Ratio of samples with a new category is not greater than 0%
        19: NewLabelTrainTest
                        0: Number of new label values is not greater than 0
        20: StringMismatchComparison
                        0: No new variants allowed in test data
        21: DatasetsSizeComparison
                        0: Test-Train size ratio is not smaller than 0.01
        22: DateTrainTestLeakageDuplicates
                        0: Date leakage ratio is not greater than 0%
        23: DateTrainTestLeakageOverlap
                        0: Date leakage ratio is not greater than 0%
        24: SingleFeatureContributionTrainTest(ppscore_params={})
                        0: Train-Test features' Predictive Power Score difference is not greater than 0.2
                        1: Train features' Predictive Power Score is not greater than 0.7
        25: TrainTestSamplesMix
                        0: Percentage of test data samples that appear in train data not greater than 10%
        26: IdentifierLeakage(ppscore_params={})
                        0: Identifier columns PPS is not greater than 0
        27: IndexTrainTestLeakage
                        0: Ratio of leaking indices is not greater than 0%
        28: IsSingleValue
                        0: Does not contain only a single value
        29: MixedNulls
                        0: Not more than 1 different null types
        30: MixedDataTypes
                        0: Rare data types in column are either more than 10% or less than 1% of the data
        31: StringMismatch
                        0: No string variants
        32: DataDuplicates
                        0: Duplicate data ratio is not greater than 0%
        33: StringLengthOutOfBounds
                        0: Ratio of outliers not greater than 0% string length outliers
        34: SpecialCharacters
                        0: Ratio of entirely special character samples not greater than 0.1%
        35: LabelAmbiguity
                        0: Ambiguous sample ratio is not greater than 0%
# now we can use the check's index and the condition's number to remove it:
                0: The performance difference of the detected segments must not be greater than 5%
# print and see that the condition was removed

If we now re-run the suite, all of the existing conditions will pass.

Note: the check we manipulated will still run as part of the Suite, however it won’t appear in the Conditions Summary since it no longer has any conditions defined on it. You can still see its display results in the Additional Outputs section

For more info about working with conditions, see the detailed configuring conditions guide.