CustomClassificationScorer

class CustomClassificationScorer

Scorer that runs a custom metric for the vision classification task.

Custom scorers can be passed to all model evaluation related checks as can be seen in the example below.

Parameters

scorert.Union[str, t.Callable]

sklearn scorer name or deepchecks supported string o rcallable

Returns

scorer: DeepcheckScorer

An initialized DeepcheckScorer.

Examples

>>> from sklearn.metrics import make_scorer, cohen_kappa_score
... from deepchecks.vision.metrics_utils.custom_scorer import CustomClassificationScorer
... from deepchecks.vision.checks.model_evaluation import SingleDatasetPerformance
... from deepchecks.vision.datasets.classification import mnist_torch as mnist
...
... mnist_model = mnist.load_model()
... test_ds = mnist.load_dataset(root='Data', object_type='VisionData')
...
>>> ck = CustomClassificationScorer(make_scorer(cohen_kappa_score))
...
>>> check = SingleDatasetPerformance(scorers={'cohen_kappa_score': ck})
... check.run(test_ds, mnist_model).value

__init__(scorer: Union[Callable, str])

__new__(*args, **kwargs)

Attributes

CustomClassificationScorer.required_output_keys

Methods

CustomClassificationScorer.attach(engine, name)	Attaches current metric to provided engine.
CustomClassificationScorer.completed(engine, ...)	Helper method to compute metric's value and put into the engine.
CustomClassificationScorer.compute()	Compute metric value.
CustomClassificationScorer.detach(engine[, ...])	Detaches current metric from the engine and no metric's computation is done during the run.
CustomClassificationScorer.is_attached(engine)	Checks if current metric is attached to provided engine.
CustomClassificationScorer.iteration_completed(engine)	Helper method to update metric's computation.
CustomClassificationScorer.reset()	Reset metric state.
CustomClassificationScorer.started(engine)	Helper method to start data gathering for metric's computation.
CustomClassificationScorer.update(output)	Update metric with batch of samples.

previous

AVAILABLE_EVALUATING_FUNCTIONS

next

CustomClassificationScorer.required_output_keys