deepchecks.vision#

Package for vision functionality.

Modules

checks

Module importing all vision checks.

suites

Module contains all prebuilt vision suites.

datasets

Module containing datasets and models for vision tasks.

utils

Package for vision utilities.

metrics_utils

Module containing metrics utils.

vision_data

Package for vision data class and utilities.

base_checks

Module for vision base checks.

context

Module for base vision context.

deprecation_warnings

This file changes default 'ignore' action of DeprecationWarnings for specific deprecation messages.

metrics

Module containing metrics implemented by Deepchecks.

suite

Module for base vision abstractions.

Classes

class VisionData[source]#

VisionData is the base data object of deepcheck vision used as input to checks and suites.

Parameters
batch_loader

A batch_loader which load a batch of data in an iterative manner. Batch loader batch output must be a dictionary in BatchOutputFormat format. The batch loader must provide SHUFFLED batches.

task_typestr

The task type of the data. can be one of the following: ‘classification’, ‘semantic_segmentation’, ‘object_detection’, ‘other’. For ‘other’, only image related checks (such as ImagePropertyOutliers) will be run.

label_mapDict[int, str], optional

A dictionary mapping class ids to their names.

dataset_name: str, optional

Name of the dataset to use in the displays instead of “Train” or “Test”.

reshuffle_data: bool, default=True

If True we will attempt to shuffle the batch loader. Only set this to False if the data is already shuffled.

Attributes
batch_loader

Return the batch loader used be the vision data.

has_additional_data

Return True if the batch loader contains additional_data.

has_embeddings

Return True if the batch loader contains embeddings.

has_image_identifiers

Return True if the batch loader contains image identifiers.

has_images

Return True if the batch loader contains images.

has_labels

Return True if the batch loader contains labels.

has_predictions

Return True if the batch loader contains predictions.

num_classes

Return a number of possible classes based on model proba, label map or observed classes.

number_of_images_cached

Return True if the number of images processed and whose statistics were cached.

task_type

Return True if the batch loader contains labels.

Methods

copy([reshuffle_data, batch_loader])

Create new copy of the vision data object with clean cache.

get_cache([use_class_names])

Return a dictionary of stored cache.

get_observed_classes([use_class_names])

Return a dictionary of observed classes either as class ids or as the class names.

head([num_images_to_display, show_in_window])

Show data from a single batch of this VisionData.

init_cache()

Initialize the cache.

update_cache(batch_size, numpy_labels, ...)

Update cache based on newly arrived batch.

validate()

Validate the VisionData functionalities implemented by the user and set which formatters were implemented.

__init__(batch_loader, task_type: typing_extensions.Literal[classification, object_detection, semantic_segmentation, other], label_map: Optional[Dict[int, str]] = None, dataset_name: Optional[str] = None, reshuffle_data: bool = True)[source]#
property batch_loader#

Return the batch loader used be the vision data.

copy(reshuffle_data: bool = False, batch_loader=None) VD[source]#

Create new copy of the vision data object with clean cache.

Parameters
reshuffle_data: bool, default=False

If True and the batch loader is of known type that can be shuffled, it will be shuffled.

batch_loader:

If not None, the batch loader of the new object will be set to this value.

Returns
VisionData

A copy of the vision data object with clean cache.

get_cache(use_class_names: bool = True) Dict[str, Any][source]#

Return a dictionary of stored cache.

get_observed_classes(use_class_names: bool = True) List[str][source]#

Return a dictionary of observed classes either as class ids or as the class names.

property has_additional_data: bool#

Return True if the batch loader contains additional_data.

property has_embeddings: bool#

Return True if the batch loader contains embeddings.

property has_image_identifiers: bool#

Return True if the batch loader contains image identifiers.

property has_images: bool#

Return True if the batch loader contains images.

property has_labels: bool#

Return True if the batch loader contains labels.

property has_predictions: bool#

Return True if the batch loader contains predictions.

head(num_images_to_display: int = 5, show_in_window: bool = False)[source]#

Show data from a single batch of this VisionData. Works only inside a notebook.

Parameters
num_images_to_display: int, default = 5

Number of images to show. Does not show more images than the size of single batch

show_in_window: bool, default = False

Whether to open the head display in a new python window. requires pyqt5, pyqtwebengine libraries.

init_cache()[source]#

Initialize the cache.

property num_classes: int#

Return a number of possible classes based on model proba, label map or observed classes.

property number_of_images_cached: int#

Return True if the number of images processed and whose statistics were cached.

property task_type: TaskType#

Return True if the batch loader contains labels.

update_cache(batch_size, numpy_labels, numpy_predictions)[source]#

Update cache based on newly arrived batch.

validate()[source]#

Validate the VisionData functionalities implemented by the user and set which formatters were implemented.

class BatchOutputFormat[source]#

Batch output format required by deepchecks.

Methods

clear()

copy()

fromkeys(iterable[, value])

Create a new dictionary with keys from iterable and values set to value.

get(key[, default])

Return the value for key if key is in the dictionary, else default.

items()

keys()

pop(key[, default])

If key is not found, default is returned if given, otherwise KeyError is raised

popitem(/)

Remove and return a (key, value) pair as a 2-tuple.

setdefault(key[, default])

Insert key with a value of default if key is not in the dictionary.

update([E, ]**F)

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

__init__(*args, **kwargs)#
__new__(*args, **kwargs)#
clear() None.  Remove all items from D.#
copy() a shallow copy of D#
fromkeys(iterable, value=None, /)#

Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)#

Return the value for key if key is in the dictionary, else default.

items() a set-like object providing a view on D's items#
keys() a set-like object providing a view on D's keys#
pop(key, default=<unrepresentable>, /)#

If key is not found, default is returned if given, otherwise KeyError is raised

popitem(/)#

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)#

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) None.  Update D from dict/iterable E and F.#

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values() an object providing a view on D's values#
class SingleDatasetCheck[source]#

Parent class for checks that only use one dataset.

Methods

add_condition(name, condition_func, **params)

Add new condition function to the check.

clean_conditions()

Remove all conditions from this check instance.

compute(context, dataset_kind)

Compute final check result based on accumulated internal state.

conditions_decision(result)

Run conditions on given result.

config([include_version, include_defaults])

Return check configuration (conditions' configuration not yet supported).

context_type

alias of Context

from_config(conf[, version_unmatch])

Return check object from a CheckConfig object.

from_json(conf[, version_unmatch])

Deserialize check instance from JSON string.

initialize_run(context, dataset_kind)

Initialize run before starting updating on batches.

metadata([with_doc_link])

Return check metadata.

name()

Name of class in split camel case.

params([show_defaults])

Return parameters to show when printing the check.

remove_condition(index)

Remove given condition by index.

run(dataset[, random_state, with_display])

Run check.

to_json([indent, include_version, ...])

Serialize check instance to JSON string.

update(context, batch, dataset_kind)

Update internal check state with given batch.

__init__(**kwargs)[source]#
add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#

Add new condition function to the check.

Parameters
namestr

Name of the condition. should explain the condition action and parameters

condition_funcCallable[[Any], Union[List[ConditionResult], bool]]

Function which gets the value of the check and returns object of List[ConditionResult] or boolean.

paramsdict

Additional parameters to pass when calling the condition function.

clean_conditions()[source]#

Remove all conditions from this check instance.

compute(context: Context, dataset_kind: DatasetKind) CheckResult[source]#

Compute final check result based on accumulated internal state.

conditions_decision(result: CheckResult) List[ConditionResult][source]#

Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) CheckConfig[source]#

Return check configuration (conditions’ configuration not yet supported).

Returns
CheckConfig

includes the checks class name, params, and module name.

context_type[source]#

alias of Context

classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Return check object from a CheckConfig object.

Parameters
confDict[Any, Any]
Returns
BaseCheck

the check class object from given config

classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Deserialize check instance from JSON string.

initialize_run(context: Context, dataset_kind: DatasetKind)[source]#

Initialize run before starting updating on batches. Optional.

metadata(with_doc_link: bool = False) CheckMetadata[source]#

Return check metadata.

Parameters
with_doc_linkbool, default False

whethere to include doc link in summary or not

Returns
Dict[str, Any]
classmethod name() str[source]#

Name of class in split camel case.

params(show_defaults: bool = False) Dict[source]#

Return parameters to show when printing the check.

remove_condition(index: int)[source]#

Remove given condition by index.

Parameters
indexint

index of condtion to remove

run(dataset: VisionData, random_state: int = 42, with_display: bool = True) CheckResult[source]#

Run check.

Parameters
dataset: VisionData

VisionData object to process

random_stateint

A seed to set for pseudo-random functions

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) str[source]#

Serialize check instance to JSON string.

update(context: Context, batch: Any, dataset_kind: DatasetKind)[source]#

Update internal check state with given batch.

class TrainTestCheck[source]#

Parent class for checks that compare two datasets.

The class checks train dataset and test dataset for model training and test.

Methods

add_condition(name, condition_func, **params)

Add new condition function to the check.

clean_conditions()

Remove all conditions from this check instance.

compute(context)

Compute final check result based on accumulated internal state.

conditions_decision(result)

Run conditions on given result.

config([include_version, include_defaults])

Return check configuration (conditions' configuration not yet supported).

context_type

alias of Context

from_config(conf[, version_unmatch])

Return check object from a CheckConfig object.

from_json(conf[, version_unmatch])

Deserialize check instance from JSON string.

initialize_run(context)

Initialize run before starting updating on batches.

metadata([with_doc_link])

Return check metadata.

name()

Name of class in split camel case.

params([show_defaults])

Return parameters to show when printing the check.

remove_condition(index)

Remove given condition by index.

run(train_dataset, test_dataset[, ...])

Run check.

to_json([indent, include_version, ...])

Serialize check instance to JSON string.

update(context, batch, dataset_kind)

Update internal check state with given batch for either train or test.

__init__(**kwargs)[source]#
add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#

Add new condition function to the check.

Parameters
namestr

Name of the condition. should explain the condition action and parameters

condition_funcCallable[[Any], Union[List[ConditionResult], bool]]

Function which gets the value of the check and returns object of List[ConditionResult] or boolean.

paramsdict

Additional parameters to pass when calling the condition function.

clean_conditions()[source]#

Remove all conditions from this check instance.

compute(context: Context) CheckResult[source]#

Compute final check result based on accumulated internal state.

conditions_decision(result: CheckResult) List[ConditionResult][source]#

Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) CheckConfig[source]#

Return check configuration (conditions’ configuration not yet supported).

Returns
CheckConfig

includes the checks class name, params, and module name.

context_type[source]#

alias of Context

classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Return check object from a CheckConfig object.

Parameters
confDict[Any, Any]
Returns
BaseCheck

the check class object from given config

classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Deserialize check instance from JSON string.

initialize_run(context: Context)[source]#

Initialize run before starting updating on batches. Optional.

metadata(with_doc_link: bool = False) CheckMetadata[source]#

Return check metadata.

Parameters
with_doc_linkbool, default False

whethere to include doc link in summary or not

Returns
Dict[str, Any]
classmethod name() str[source]#

Name of class in split camel case.

params(show_defaults: bool = False) Dict[source]#

Return parameters to show when printing the check.

remove_condition(index: int)[source]#

Remove given condition by index.

Parameters
indexint

index of condtion to remove

run(train_dataset: VisionData, test_dataset: VisionData, random_state: int = 42, with_display: bool = True) CheckResult[source]#

Run check.

Parameters
train_datasetVisionData

VisionData object, representing data the model was fitted on

test_datasetVisionData

VisionData object, representing data the models predicts on

random_stateint

A seed to set for pseudo-random functions

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) str[source]#

Serialize check instance to JSON string.

update(context: Context, batch: Any, dataset_kind: DatasetKind)[source]#

Update internal check state with given batch for either train or test.

class ModelOnlyCheck[source]#

Parent class for checks that only use a model and no datasets.

Methods

add_condition(name, condition_func, **params)

Add new condition function to the check.

clean_conditions()

Remove all conditions from this check instance.

compute(context)

Compute final check result.

conditions_decision(result)

Run conditions on given result.

config([include_version, include_defaults])

Return check configuration (conditions' configuration not yet supported).

context_type

alias of Context

from_config(conf[, version_unmatch])

Return check object from a CheckConfig object.

from_json(conf[, version_unmatch])

Deserialize check instance from JSON string.

initialize_run(context)

Initialize run before starting updating on batches.

metadata([with_doc_link])

Return check metadata.

name()

Name of class in split camel case.

params([show_defaults])

Return parameters to show when printing the check.

remove_condition(index)

Remove given condition by index.

run(model[, random_state, with_display])

Run check.

to_json([indent, include_version, ...])

Serialize check instance to JSON string.

__init__(**kwargs)[source]#
add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#

Add new condition function to the check.

Parameters
namestr

Name of the condition. should explain the condition action and parameters

condition_funcCallable[[Any], Union[List[ConditionResult], bool]]

Function which gets the value of the check and returns object of List[ConditionResult] or boolean.

paramsdict

Additional parameters to pass when calling the condition function.

clean_conditions()[source]#

Remove all conditions from this check instance.

compute(context: Context) CheckResult[source]#

Compute final check result.

conditions_decision(result: CheckResult) List[ConditionResult][source]#

Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) CheckConfig[source]#

Return check configuration (conditions’ configuration not yet supported).

Returns
CheckConfig

includes the checks class name, params, and module name.

context_type[source]#

alias of Context

classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Return check object from a CheckConfig object.

Parameters
confDict[Any, Any]
Returns
BaseCheck

the check class object from given config

classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Deserialize check instance from JSON string.

initialize_run(context: Context)[source]#

Initialize run before starting updating on batches. Optional.

metadata(with_doc_link: bool = False) CheckMetadata[source]#

Return check metadata.

Parameters
with_doc_linkbool, default False

whethere to include doc link in summary or not

Returns
Dict[str, Any]
classmethod name() str[source]#

Name of class in split camel case.

params(show_defaults: bool = False) Dict[source]#

Return parameters to show when printing the check.

remove_condition(index: int)[source]#

Remove given condition by index.

Parameters
indexint

index of condtion to remove

run(model, random_state: int = 42, with_display: bool = True) CheckResult[source]#

Run check.

Parameters
model

Model to run the check on

random_stateint

A seed to set for pseudo-random functions

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) str[source]#

Serialize check instance to JSON string.

class Suite[source]#

Tabular suite to run checks of types: TrainTestCheck, SingleDatasetCheck, ModelOnlyCheck.

Methods

add(check)

Add a check or a suite to current suite.

config()

Return suite configuration (checks' conditions' configuration not yet supported).

from_config(conf[, version_unmatch])

Return suite object from a CheckConfig object.

from_json(conf[, version_unmatch])

Deserialize suite instance from JSON string.

remove(index)

Remove a check by given index.

run([train_dataset, test_dataset, ...])

Run all checks.

supported_checks()

Return tuple of supported check types of this suite.

to_json([indent])

Serialize suite instance to JSON string.

__init__(name: str, *checks: Union[BaseCheck, BaseSuite])[source]#
add(check: Union[BaseCheck, BaseSuite])[source]#

Add a check or a suite to current suite.

Parameters
checkBaseCheck

A check or suite to add.

config() SuiteConfig[source]#

Return suite configuration (checks’ conditions’ configuration not yet supported).

Returns
SuiteConfig

includes the suite name, and list of check configs.

classmethod from_config(conf: ~deepchecks.core.suite.SuiteConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Return suite object from a CheckConfig object.

Parameters
confSuiteConfig

the SuiteConfig object

Returns
BaseSuite

the suite class object from given config

from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self[source]#

Deserialize suite instance from JSON string.

remove(index: int)[source]#

Remove a check by given index.

Parameters
indexint

Index of check to remove.

run(train_dataset: Optional[VisionData] = None, test_dataset: Optional[VisionData] = None, random_state: int = 42, with_display: bool = True, max_samples: Optional[int] = None, run_single_dataset: Optional[str] = None) SuiteResult[source]#

Run all checks.

Parameters
train_datasetOptional[VisionData] , default: None

VisionData object, representing data the model was fitted on

test_datasetOptional[VisionData] , default: None

VisionData object, representing data the models predicts on

random_stateint

A seed to set for pseudo-random functions

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

max_samplesOptional[int] , default: None

Each check will run on a number of samples which is the minimum between the n_samples parameter of the check and this parameter. If this argument is None then the number of samples for each check will be determined by the n_samples argument.

run_single_dataset: Optional[str], default None

‘Train’, ‘Test’ , or None to run on both train and test.

Returns
SuiteResult

All results by all initialized checks

classmethod supported_checks() Tuple[source]#

Return tuple of supported check types of this suite.

to_json(indent: int = 3) str[source]#

Serialize suite instance to JSON string.

Functions

classification_dataset_from_directory(root: str, batch_size: int = 32, num_workers: int = 0, shuffle: bool = True, pin_memory: bool = True, object_type: typing_extensions.Literal[VisionData, DataLoader] = 'DataLoader', **kwargs) Union[Tuple[Union[DataLoader, VisionData]], DataLoader, VisionData][source]#

Load a simple classification dataset.

The function expects that the data within the root folder to be structured one of the following ways:

  • root/
    • class1/

      image1.jpeg

  • root/
    • train/
      • class1/

        image1.jpeg

    • test/
      • class1/

        image1.jpeg

Parameters
rootstr

path to the data

batch_sizeint, default: 32

Batch size for the dataloader.

num_workersint, default: 0

Number of workers for the dataloader.

shufflebool, default: True

Whether to shuffle the dataset.

pin_memorybool, default: True

If True, the data loader will copy Tensors into CUDA pinned memory before returning them.

object_typeLiteral[‘Dataset’, ‘DataLoader’], default: ‘DataLoader’

type of the return value. If ‘Dataset’, deepchecks.vision.VisionData will be returned, otherwise torch.utils.data.DataLoader

Returns
t.Union[t.Tuple[t.Union[DataLoader, vision.ClassificationData]], t.Union[DataLoader, vision.ClassificationData]]

A DataLoader or VisionDataset instance or tuple representing a single dataset or train and test datasets.