TrainTestPredictionDrift#
- class TrainTestPredictionDrift[source]#
Calculate prediction drift between train dataset and test dataset, using statistical measures.
Check calculates a drift score for the prediction in the test dataset, by comparing its distribution to the train dataset. For classification tasks, by default the drift score will be computed on the predicted probability of the positive (1) class for binary classification tasks, and on the predicted class itself for multiclass tasks. This behavior can be controlled using the drift_mode parameter.
For numerical columns, we use the Earth Movers Distance. See https://en.wikipedia.org/wiki/Wasserstein_metric
For categorical distributions, we use the Cramer’s V. See https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V We also support Population Stability Index (PSI). See https://www.lexjansen.com/wuss/2017/47_Final_Paper_PDF.pdf.
For categorical predictions, it is recommended to use Cramer’s V, unless your variable includes categories with a small number of samples (common practice is categories with less than 5 samples). However, in cases of a variable with many categories with few samples, it is still recommended to use Cramer’s V.
- Parameters
- drift_mode: str, default: ‘auto’
For classification task, controls whether to compute drift on the predicted probabilities or the predicted classes. For regression task this parameter may be ignored. If set to ‘auto’, compute drift on the predicted class if the task is multiclass, and on the predicted probability of the positive class if binary. Set to ‘proba’ to force drift on the predicted probabilities, and ‘prediction’ to force drift on the predicted classes. If set to ‘proba’, on a multiclass task, drift would be calculated on each class independently.
- margin_quantile_filter: float, default: 0.025
float in range [0,0.5), representing which margins (high and low quantiles) of the distribution will be filtered out of the EMD calculation. This is done in order for extreme values not to affect the calculation disproportionally. This filter is applied to both distributions, in both margins.
- min_category_size_ratio: float, default 0.01
minimum size ratio for categories. Categories with size ratio lower than this number are binned into an “Other” category.
- max_num_categories_for_drift: int, default: None
Only relevant if drift is calculated for classification predictions. Max number of allowed categories. If there are more, they are binned into an “Other” category. This limit applies for both drift calculation and distribution plots.
- max_num_categories_for_display: int, default: 10
Max number of categories to show in plot.
- show_categories_by: str, default: ‘largest_difference’
Specify which categories to show for categorical features’ graphs, as the number of shown categories is limited by max_num_categories_for_display. Possible values: - ‘train_largest’: Show the largest train categories. - ‘test_largest’: Show the largest test categories. - ‘largest_difference’: Show the largest difference between categories.
- categorical_drift_method: str, default: “cramer_v”
decides which method to use on categorical variables. Possible values are: “cramer_v” for Cramer’s V, “PSI” for Population Stability Index (PSI).
- ignore_na: bool, default True
For categorical columns only. If True, ignores nones for categorical drift. If False, considers none as a separate category. For numerical columns we always ignore nones.
- aggregation_method: str, default: “max”
Argument for the reduce_output functionality, decides how to aggregate the drift scores of different classes (for classification tasks) into a single score, when drift is computed on the class probabilities. Possible values are: ‘max’: Maximum of all the class drift scores. ‘weighted’: Weighted mean based on the class sizes in the train data set. ‘mean’: Mean of all drift scores. ‘none’: No averaging. Return a dict with a drift score for each class.
- max_classes_to_display: int, default: 3
Max number of classes to show in the display when drift is computed on the class probabilities for classification tasks.
- n_samplesint , default: 100_000
number of samples to use for this check.
- random_stateint, default: 42
random seed for all check internals.
- __init__(drift_mode: str = 'auto', margin_quantile_filter: float = 0.025, max_num_categories_for_drift: Optional[int] = None, min_category_size_ratio: float = 0.01, max_num_categories_for_display: int = 10, show_categories_by: str = 'largest_difference', categorical_drift_method: str = 'cramer_v', ignore_na: bool = True, aggregation_method: str = 'max', max_classes_to_display: int = 3, n_samples: int = 100000, random_state: int = 42, **kwargs)[source]#
- __new__(*args, **kwargs)#
Methods
|
Add new condition function to the check. |
|
Add condition - require drift score to be less than a certain threshold. |
Remove all conditions from this check instance. |
|
Run conditions on given result. |
|
Return check configuration (conditions' configuration not yet supported). |
|
|
Return check object from a CheckConfig object. |
|
Deserialize check instance from JSON string. |
Return True if the check reduce_output is better when it is greater. |
|
Return check metadata. |
|
Name of class in split camel case. |
|
|
Return parameters to show when printing the check. |
Return prediction drift score. |
|
Remove given condition by index. |
|
|
Run check. |
|
Calculate drift for all columns. |
|
Serialize check instance to JSON string. |
