CustomClassificationScorer#

class CustomClassificationScorer[source]#

Scorer that runs a custom metric for the vision classification task.

Custom scorers can be passed to all model evaluation related checks as can be seen in the example below.

Parameters
scorert.Union[str, t.Callable]

sklearn scorer name or deepchecks supported string o rcallable

Returns
scorer: DeepcheckScorer

An initialized DeepcheckScorer.

Examples

>>> from sklearn.metrics import make_scorer, cohen_kappa_score
... from deepchecks.vision.metrics_utils.custom_scorer import CustomClassificationScorer
... from deepchecks.vision.checks.model_evaluation import SingleDatasetPerformance
... from deepchecks.vision.datasets.classification import mnist_torch as mnist
...
... mnist_model = mnist.load_model()
... test_ds = mnist.load_dataset(root='Data', object_type='VisionData')
...
>>> ck = CustomClassificationScorer(make_scorer(cohen_kappa_score))
...
>>> check = SingleDatasetPerformance(scorers={'cohen_kappa_score': ck})
... check.run(test_ds, mnist_model).value
__init__(scorer: Union[Callable, str])[source]#
__new__(*args, **kwargs)#

Attributes

CustomClassificationScorer.required_output_keys

Methods

CustomClassificationScorer.attach(engine, name)

Attaches current metric to provided engine.

CustomClassificationScorer.completed(engine, ...)

Helper method to compute metric's value and put into the engine.

CustomClassificationScorer.compute()

Compute metric value.

CustomClassificationScorer.detach(engine[, ...])

Detaches current metric from the engine and no metric's computation is done during the run.

CustomClassificationScorer.is_attached(engine)

Checks if current metric is attached to provided engine.

CustomClassificationScorer.iteration_completed(engine)

Helper method to update metric's computation.

CustomClassificationScorer.load_state_dict(...)

Method replaces internal state of the class with provided state dict data.

CustomClassificationScorer.reset()

Reset metric state.

CustomClassificationScorer.started(engine)

Helper method to start data gathering for metric's computation.

CustomClassificationScorer.state_dict()

Method returns state dict with attributes of the metric specified in its _state_dict_all_req_keys attribute.

CustomClassificationScorer.update(output)

Update metric with batch of samples.

previous

AVAILABLE_EVALUATING_FUNCTIONS

next

CustomClassificationScorer.required_output_keys