Create a Custom Suite#

A suite is a list of checks that will run one after the other, and its results will be displayed together.

To customize a suite, we can either:

Create a New Suite#

Let’s say we want to create our custom suite, mainly with various performance checks, including PerformanceReport(), TrainTestDifferenceOverfit() and several more.

For assistance in understanding which checks are implemented and can be included, we suggest using any of:

from sklearn.metrics import make_scorer, precision_score, recall_score

from deepchecks.tabular import Suite
# importing all existing checks for demonstration simplicity
from deepchecks.tabular.checks import *

# The Suite's first argument is its name, and then all of the check objects.
# Some checks can receive arguments when initialized (all check arguments have default values)
# Each check can have an optional condition(/s)
# Multiple conditions can be applied subsequentially
new_custom_suite = Suite('Simple Suite For Model Performance',
    ModelInfo(),
    # use custom scorers for performance report:
    PerformanceReport().add_condition_train_test_relative_degradation_not_greater_than(threshold=0.15\
                     ).add_condition_test_performance_not_less_than(0.8),
    ConfusionMatrixReport(),
    SimpleModelComparison(simple_model_type='constant', \
                          alternative_scorers={'Recall (Multiclass)': make_scorer(recall_score, average=None), \
                                               'Precision (Multiclass)': make_scorer(precision_score, average=None)} \
                         ).add_condition_gain_greater_than(0.3)
    )
# Let's see the suite:
new_custom_suite

Out:

Simple Suite For Model Performance: [
    0: ModelInfo
    1: PerformanceReport
            Conditions:
                    0: Train-Test scores relative degradation is not greater than 0.15
                    1: Scores are not less than 0.8
    2: ConfusionMatrixReport
    3: SimpleModelComparison
            Conditions:
                    0: Model performance gain over simple model is greater than 30%
]

TIP: the auto-complete may not work from inside a new suite definition, so if you want to use the auto-complete to see the arguments a check receive or the built-in conditions it has, try doing it outside of the suite’s initialization.

For example, to see a check’s built-in conditions, type in a new cell: ``NameOfDesiredCheck().add_condition_`` and then check the auto-complete suggestions (using Shift + Tab), to discover the built-in checks.

Additional Notes about Conditions in a Suite#

  • Checks in the built-in suites come with pre-defined conditions, and when building your custom suite you should choose which conditions to add.

  • Most check classes have built-in methods for adding monditions. These apply to the naming convention add_condition_..., which enables adding a condition logic to parse the check’s results.

  • Each check instance can have several conditions or none. Each condition will be evaluated separately.

  • The pass (✓) / fail (✖) / insight (!) status of the conditions, along with the condition’s name and extra info will be displayed in the suite’s Conditions Summary.

  • Most conditions have configurable arguments that can be passed to the condition while adding it.

  • For more info about conditions, check out Configure a Condition.

Run the Suite#

This is simply done by calling the run() method of the suite.

To see that in action, we’ll need datasets and a model.

Let’s quickly load a dataset and train a simple model for the sake of this demo

Load Datasets and Train a Simple Model#

import numpy as np
# General imports
import pandas as pd

np.random.seed(22)

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from deepchecks.tabular.datasets.classification import iris

# Load pre-split Datasets
train_dataset, test_dataset = iris.load_data(as_train_test=True)
label_col = 'target'

# Train Model
rf_clf = RandomForestClassifier()
rf_clf.fit(train_dataset.data[train_dataset.features],
           train_dataset.data[train_dataset.label_name]);

Out:

RandomForestClassifier()

Run Suite#

new_custom_suite.run(model=rf_clf, train_dataset=train_dataset, test_dataset=test_dataset)

Out:

Simple Suite For Model Performance:
|     | 0/4 [00:00<?, ? Check/s]
Simple Suite For Model Performance:
|##5  | 2/4 [00:00<00:00,  8.09 Check/s, Check=Performance Report]Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.

Simple Suite For Model Performance:
|#####| 4/4 [00:00<00:00,  9.41 Check/s, Check=Simple Model Comparison]
Simple Suite For Model Performance


Modify an Existing Suite#

from deepchecks.tabular.suites import train_test_leakage

customized_suite = train_test_leakage()

# let's check what it has:
customized_suite

Out:

the train_test_leakage suite is deprecated, use the train_test_validation suite instead

Train Test Validation Suite: [
    0: DatasetsSizeComparison
            Conditions:
                    0: Test-Train size ratio is greater than 0.01
    1: NewLabelTrainTest
            Conditions:
                    0: Number of new label values is less or equal to 0
    2: CategoryMismatchTrainTest
            Conditions:
                    0: Ratio of samples with a new category is less or equal to 0%
    3: StringMismatchComparison
            Conditions:
                    0: No new variants allowed in test data
    4: DateTrainTestLeakageDuplicates
            Conditions:
                    0: Date leakage ratio is less or equal to 0%
    5: DateTrainTestLeakageOverlap
            Conditions:
                    0: Date leakage ratio is less or equal to 0%
    6: IndexTrainTestLeakage
            Conditions:
                    0: Ratio of leaking indices is less or equal to 0%
    7: IdentifierLeakage(ppscore_params={})
            Conditions:
                    0: Identifier columns PPS is less or equal to 0
    8: TrainTestSamplesMix
            Conditions:
                    0: Percentage of test data samples that appear in train data is less or equal to 10%
    9: FeatureLabelCorrelationChange(ppscore_params={})
            Conditions:
                    0: Train-Test features' Predictive Power Score difference is less than 0.2
                    1: Train features' Predictive Power Score is less than 0.7
    10: TrainTestFeatureDrift
            Conditions:
                    0: categorical drift score < 0.2 and numerical drift score < 0.1
    11: TrainTestLabelDrift
            Conditions:
                    0: categorical drift score < 0.2 and numerical drift score < 0.1 for label drift
    12: WholeDatasetDrift
            Conditions:
                    0: Drift value is less than 0.25
]
# and modify it by removing a check by index:
customized_suite.remove(1)

Out:

Train Test Validation Suite: [
    0: DatasetsSizeComparison
            Conditions:
                    0: Test-Train size ratio is greater than 0.01
    2: CategoryMismatchTrainTest
            Conditions:
                    0: Ratio of samples with a new category is less or equal to 0%
    3: StringMismatchComparison
            Conditions:
                    0: No new variants allowed in test data
    4: DateTrainTestLeakageDuplicates
            Conditions:
                    0: Date leakage ratio is less or equal to 0%
    5: DateTrainTestLeakageOverlap
            Conditions:
                    0: Date leakage ratio is less or equal to 0%
    6: IndexTrainTestLeakage
            Conditions:
                    0: Ratio of leaking indices is less or equal to 0%
    7: IdentifierLeakage(ppscore_params={})
            Conditions:
                    0: Identifier columns PPS is less or equal to 0
    8: TrainTestSamplesMix
            Conditions:
                    0: Percentage of test data samples that appear in train data is less or equal to 10%
    9: FeatureLabelCorrelationChange(ppscore_params={})
            Conditions:
                    0: Train-Test features' Predictive Power Score difference is less than 0.2
                    1: Train features' Predictive Power Score is less than 0.7
    10: TrainTestFeatureDrift
            Conditions:
                    0: categorical drift score < 0.2 and numerical drift score < 0.1
    11: TrainTestLabelDrift
            Conditions:
                    0: categorical drift score < 0.2 and numerical drift score < 0.1 for label drift
    12: WholeDatasetDrift
            Conditions:
                    0: Drift value is less than 0.25
]
from deepchecks.tabular.checks import UnusedFeatures

# and add a new check with a condition:
customized_suite.add(
    UnusedFeatures().add_condition_number_of_high_variance_unused_features_less_or_equal())

Out:

Train Test Validation Suite: [
    0: DatasetsSizeComparison
            Conditions:
                    0: Test-Train size ratio is greater than 0.01
    2: CategoryMismatchTrainTest
            Conditions:
                    0: Ratio of samples with a new category is less or equal to 0%
    3: StringMismatchComparison
            Conditions:
                    0: No new variants allowed in test data
    4: DateTrainTestLeakageDuplicates
            Conditions:
                    0: Date leakage ratio is less or equal to 0%
    5: DateTrainTestLeakageOverlap
            Conditions:
                    0: Date leakage ratio is less or equal to 0%
    6: IndexTrainTestLeakage
            Conditions:
                    0: Ratio of leaking indices is less or equal to 0%
    7: IdentifierLeakage(ppscore_params={})
            Conditions:
                    0: Identifier columns PPS is less or equal to 0
    8: TrainTestSamplesMix
            Conditions:
                    0: Percentage of test data samples that appear in train data is less or equal to 10%
    9: FeatureLabelCorrelationChange(ppscore_params={})
            Conditions:
                    0: Train-Test features' Predictive Power Score difference is less than 0.2
                    1: Train features' Predictive Power Score is less than 0.7
    10: TrainTestFeatureDrift
            Conditions:
                    0: categorical drift score < 0.2 and numerical drift score < 0.1
    11: TrainTestLabelDrift
            Conditions:
                    0: categorical drift score < 0.2 and numerical drift score < 0.1 for label drift
    12: WholeDatasetDrift
            Conditions:
                    0: Drift value is less than 0.25
    13: UnusedFeatures
            Conditions:
                    0: Number of high variance unused features is less or equal to 5
]
# lets remove all condition for the FeatureLabelCorrelationChange:
customized_suite[3].clean_conditions()

# and update the suite's name:
customized_suite.name = 'New Data Leakage Suite'
# and now we can run our modified suite:
customized_suite.run(train_dataset, test_dataset, rf_clf)

Out:

New Data Leakage Suite:
|             | 0/13 [00:00<?, ? Check/s]
New Data Leakage Suite:
|#########    | 9/13 [00:00<00:00, 82.49 Check/s, Check=Feature Label Correlation Change]
New Data Leakage Suite


Total running time of the script: ( 0 minutes 3.196 seconds)

Gallery generated by Sphinx-Gallery