SimpleModelComparison#
- class SimpleModelComparison[source]#
Compare given model score to simple model score (according to given model type).
For classification models, the simple model is a dummy classifier the selects the predictions based on a strategy.
- Parameters
- strategystr, default=’prior’
Strategy to use to generate the predictions of the simple model.
‘most_frequent’ : The most frequent label in the training set is predicted. The probability vector is 1 for the most frequent label and 0 for the other predictions.
‘prior’ : The probability vector always contains the empirical class prior distribution (i.e. the class distribution observed in the training set).
‘stratified’ : The predictions are generated by sampling one-hot vectors from a multinomial distribution parametrized by the empirical class prior probabilities.
‘uniform’ : Generates predictions uniformly at random from the list of unique classes observed in y, i.e. each class has equal probability. The predicted class is chosen randomly.
- scorers: Union[Dict[str, Union[Callable, str]], List[Any]], default: None
Scorers to override the default scorers (metrics), find more about the supported formats at https://docs.deepchecks.com/stable/user-guide/general/metrics_guide.html
- n_to_showint, default: 20
Number of classes to show in the report. If None, show all classes.
- show_onlystr, default: ‘largest’
Specify which classes to show in the report. Can be one of the following: - ‘largest’: Show the largest classes. - ‘smallest’: Show the smallest classes. - ‘random’: Show random classes. - ‘best’: Show the classes with the highest score. - ‘worst’: Show the classes with the lowest score.
- metric_to_show_bystr, default: None
Specify the metric to sort the results by. Relevant only when show_only is ‘best’ or ‘worst’. If None, sorting by the first metric in the default metrics list.
- class_list_to_show: List[int], default: None
Specify the list of classes to show in the report. If specified, n_to_show, show_only and metric_to_show_by are ignored.
- n_samplesOptional[int] , default10000
Number of samples to use for the check. If None, all samples will be used.
- __init__(scorers: Optional[Union[Dict[str, Union[Callable, str]], List[Any]]] = None, strategy: str = 'most_frequent', n_to_show: int = 20, show_only: str = 'largest', metric_to_show_by: Optional[str] = None, class_list_to_show: Optional[List[int]] = None, n_samples: Optional[int] = 10000, **kwargs)[source]#
- __new__(*args, **kwargs)#
Methods
|
Add new condition function to the check. |
|
Add condition - require gain between the model and the simple model to be greater than threshold. |
Remove all conditions from this check instance. |
|
|
Compute the metrics for the check. |
Run conditions on given result. |
|
|
Return check configuration (conditions' configuration not yet supported). |
|
Return check object from a CheckConfig object. |
|
Deserialize check instance from JSON string. |
|
Initialize the metrics for the check, and validate task type is relevant. |
|
Return check metadata. |
Name of class in split camel case. |
|
|
Return parameters to show when printing the check. |
Remove given condition by index. |
|
|
Run check. |
|
Serialize check instance to JSON string. |
|
Update the metrics for the check. |