deepchecks.vision#

Package for vision functionality.

Modules

`checks`	Module importing all vision checks.
`suites`	Module contains all prebuilt vision suites.
`datasets`	Module containing datasets and models for vision tasks.
`utils`	Package for vision utilities.
`metrics_utils`	Module containing metrics utils.
`vision_data`	Package for vision data class and utilities.
`base_checks`	Module for vision base checks.
`context`	Module for base vision context.
`deprecation_warnings`	This file changes default 'ignore' action of DeprecationWarnings for specific deprecation messages.
`metrics`	Module containing metrics implemented by Deepchecks.
`suite`	Module for base vision abstractions.

Classes

class VisionData[source]#

VisionData is the base data object of deepcheck vision used as input to checks and suites.

Parameters

batch_loader: A batch_loader which load a batch of data in an iterative manner. Batch loader batch output must be a dictionary in BatchOutputFormat format. The batch loader must provide SHUFFLED batches.
task_typestr: The task type of the data. can be one of the following: ‘classification’, ‘semantic_segmentation’, ‘object_detection’, ‘other’. For ‘other’, only image related checks (such as ImagePropertyOutliers) will be run.
label_mapDict[int, str], optional: A dictionary mapping class ids to their names.
dataset_name: str, optional: Name of the dataset to use in the displays instead of “Train” or “Test”.
reshuffle_data: bool, default=True: If True we will attempt to shuffle the batch loader. Only set this to False if the data is already shuffled.

Attributes

batch_loader: Return the batch loader used be the vision data.
has_additional_data: Return True if the batch loader contains additional_data.
has_embeddings: Return True if the batch loader contains embeddings.
has_image_identifiers: Return True if the batch loader contains image identifiers.
has_images: Return True if the batch loader contains images.
has_labels: Return True if the batch loader contains labels.
has_predictions: Return True if the batch loader contains predictions.
num_classes: Return a number of possible classes based on model proba, label map or observed classes.
number_of_images_cached: Return True if the number of images processed and whose statistics were cached.
task_type: Return True if the batch loader contains labels.

Methods

`copy`([reshuffle_data, batch_loader])	Create new copy of the vision data object with clean cache.
`get_cache`([use_class_names])	Return a dictionary of stored cache.
`get_observed_classes`([use_class_names])	Return a dictionary of observed classes either as class ids or as the class names.
`head`([num_images_to_display, show_in_window])	Show data from a single batch of this VisionData.
`init_cache`()	Initialize the cache.
`update_cache`(batch_size, numpy_labels, ...)	Update cache based on newly arrived batch.
`validate`()	Validate the VisionData functionalities implemented by the user and set which formatters were implemented.

__init__(batch_loader, task_type: Literal['classification', 'object_detection', 'semantic_segmentation', 'other'], label_map: Optional[Dict[int, str]] = None, dataset_name: Optional[str] = None, reshuffle_data: bool = True)[source]#

property batch_loader#: Return the batch loader used be the vision data.

copy(reshuffle_data: bool = False, batch_loader=None) → VD[source]#

Create new copy of the vision data object with clean cache.

Parameters

reshuffle_data: bool, default=False: If True and the batch loader is of known type that can be shuffled, it will be shuffled.
batch_loader:: If not None, the batch loader of the new object will be set to this value.

Returns

VisionData: A copy of the vision data object with clean cache.

get_cache(use_class_names: bool = True) → Dict[str, Any][source]#: Return a dictionary of stored cache.

get_observed_classes(use_class_names: bool = True) → List[str][source]#: Return a dictionary of observed classes either as class ids or as the class names.

property has_additional_data: bool#: Return True if the batch loader contains additional_data.

property has_embeddings: bool#: Return True if the batch loader contains embeddings.

property has_image_identifiers: bool#: Return True if the batch loader contains image identifiers.

property has_images: bool#: Return True if the batch loader contains images.

property has_labels: bool#: Return True if the batch loader contains labels.

property has_predictions: bool#: Return True if the batch loader contains predictions.

head(num_images_to_display: int = 5, show_in_window: bool = False)[source]#

Show data from a single batch of this VisionData. Works only inside a notebook.

Parameters

num_images_to_display: int, default = 5: Number of images to show. Does not show more images than the size of single batch
show_in_window: bool, default = False: Whether to open the head display in a new python window. requires pyqt5, pyqtwebengine libraries.

init_cache()[source]#: Initialize the cache.

property num_classes: int#: Return a number of possible classes based on model proba, label map or observed classes.

property number_of_images_cached: int#: Return True if the number of images processed and whose statistics were cached.

property task_type: TaskType#: Return True if the batch loader contains labels.

update_cache(batch_size, numpy_labels, numpy_predictions)[source]#: Update cache based on newly arrived batch.

validate()[source]#: Validate the VisionData functionalities implemented by the user and set which formatters were implemented.

class BatchOutputFormat[source]#

Batch output format required by deepchecks.

Methods

`clear`()
`copy`()
`fromkeys`(iterable[, value])	Create a new dictionary with keys from iterable and values set to value.
`get`(key[, default])	Return the value for key if key is in the dictionary, else default.
`items`()
`keys`()
`pop`(key[, default])	If key is not found, default is returned if given, otherwise KeyError is raised
`popitem`(/)	Remove and return a (key, value) pair as a 2-tuple.
`setdefault`(key[, default])	Insert key with a value of default if key is not in the dictionary.
`update`([E, ]**F)	If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
`values`()

__init__(*args, **kwargs)#

static __new__($cls, _typename, _fields=None, **kwargs)#

clear() → None. Remove all items from D.#

copy() → a shallow copy of D#

fromkeys(iterable, value=None, /)#: Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)#: Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D's items#

keys() → a set-like object providing a view on D's keys#

pop(key, default=<unrepresentable>, /)#: If key is not found, default is returned if given, otherwise KeyError is raised

popitem(/)#

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)#

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None. Update D from dict/iterable E and F.#: If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values() → an object providing a view on D's values#

class SingleDatasetCheck[source]#

Parent class for checks that only use one dataset.

Methods

`add_condition`(name, condition_func, **params)	Add new condition function to the check.
`clean_conditions`()	Remove all conditions from this check instance.
`compute`(context, dataset_kind)	Compute final check result based on accumulated internal state.
`conditions_decision`(result)	Run conditions on given result.
`config`([include_version, include_defaults])	Return check configuration (conditions' configuration not yet supported).
`context_type`	alias of `Context`
`from_config`(conf[, version_unmatch])	Return check object from a CheckConfig object.
`from_json`(conf[, version_unmatch])	Deserialize check instance from JSON string.
`initialize_run`(context, dataset_kind)	Initialize run before starting updating on batches.
`metadata`([with_doc_link])	Return check metadata.
`name`()	Name of class in split camel case.
`params`([show_defaults])	Return parameters to show when printing the check.
`remove_condition`(index)	Remove given condition by index.
`run`(dataset[, random_state, with_display])	Run check.
`to_json`([indent, include_version, ...])	Serialize check instance to JSON string.
`update`(context, batch, dataset_kind)	Update internal check state with given batch.

__init__(**kwargs)[source]#

add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#

Add new condition function to the check.

Parameters

namestr: Name of the condition. should explain the condition action and parameters
condition_funcCallable[[Any], Union[List[ConditionResult], bool]]: Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
paramsdict: Additional parameters to pass when calling the condition function.

clean_conditions()[source]#: Remove all conditions from this check instance.

compute(context: Context, dataset_kind: DatasetKind) → CheckResult[source]#: Compute final check result based on accumulated internal state.

conditions_decision(result: CheckResult) → List[ConditionResult][source]#: Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) → CheckConfig[source]#

Return check configuration (conditions’ configuration not yet supported).

Returns

CheckConfig: includes the checks class name, params, and module name.

context_type[source]#: alias of Context

classmethod from_config(conf: CheckConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#

Return check object from a CheckConfig object.

Parameters

confDict[Any, Any]

Returns

BaseCheck: the check class object from given config

classmethod from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#: Deserialize check instance from JSON string.

initialize_run(context: Context, dataset_kind: DatasetKind)[source]#: Initialize run before starting updating on batches. Optional.

metadata(with_doc_link: bool = False) → CheckMetadata[source]#

Return check metadata.

Parameters

with_doc_linkbool, default False: whethere to include doc link in summary or not

Returns

Dict[str, Any]

classmethod name() → str[source]#: Name of class in split camel case.

params(show_defaults: bool = False) → Dict[source]#: Return parameters to show when printing the check.

remove_condition(index: int)[source]#

Remove given condition by index.

Parameters

indexint: index of condtion to remove

run(dataset: VisionData, random_state: int = 42, with_display: bool = True) → CheckResult[source]#

Run check.

Parameters

dataset: VisionData: VisionData object to process
random_stateint: A seed to set for pseudo-random functions
with_displaybool , default: True: flag that determines if checks will calculate display (redundant in some checks).

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) → str[source]#: Serialize check instance to JSON string.

update(context: Context, batch: Any, dataset_kind: DatasetKind)[source]#: Update internal check state with given batch.

class TrainTestCheck[source]#

Parent class for checks that compare two datasets.

The class checks train dataset and test dataset for model training and test.

Methods

`add_condition`(name, condition_func, **params)	Add new condition function to the check.
`clean_conditions`()	Remove all conditions from this check instance.
`compute`(context)	Compute final check result based on accumulated internal state.
`conditions_decision`(result)	Run conditions on given result.
`config`([include_version, include_defaults])	Return check configuration (conditions' configuration not yet supported).
`context_type`	alias of `Context`
`from_config`(conf[, version_unmatch])	Return check object from a CheckConfig object.
`from_json`(conf[, version_unmatch])	Deserialize check instance from JSON string.
`initialize_run`(context)	Initialize run before starting updating on batches.
`metadata`([with_doc_link])	Return check metadata.
`name`()	Name of class in split camel case.
`params`([show_defaults])	Return parameters to show when printing the check.
`remove_condition`(index)	Remove given condition by index.
`run`(train_dataset, test_dataset[, ...])	Run check.
`to_json`([indent, include_version, ...])	Serialize check instance to JSON string.
`update`(context, batch, dataset_kind)	Update internal check state with given batch for either train or test.

__init__(**kwargs)[source]#

add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#

Add new condition function to the check.

Parameters

namestr: Name of the condition. should explain the condition action and parameters
condition_funcCallable[[Any], Union[List[ConditionResult], bool]]: Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
paramsdict: Additional parameters to pass when calling the condition function.

clean_conditions()[source]#: Remove all conditions from this check instance.

compute(context: Context) → CheckResult[source]#: Compute final check result based on accumulated internal state.

conditions_decision(result: CheckResult) → List[ConditionResult][source]#: Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) → CheckConfig[source]#

Return check configuration (conditions’ configuration not yet supported).

Returns

CheckConfig: includes the checks class name, params, and module name.

context_type[source]#: alias of Context

classmethod from_config(conf: CheckConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#

Return check object from a CheckConfig object.

Parameters

confDict[Any, Any]

Returns

BaseCheck: the check class object from given config

classmethod from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#: Deserialize check instance from JSON string.

initialize_run(context: Context)[source]#: Initialize run before starting updating on batches. Optional.

metadata(with_doc_link: bool = False) → CheckMetadata[source]#

Return check metadata.

Parameters

with_doc_linkbool, default False: whethere to include doc link in summary or not

Returns

Dict[str, Any]

classmethod name() → str[source]#: Name of class in split camel case.

params(show_defaults: bool = False) → Dict[source]#: Return parameters to show when printing the check.

remove_condition(index: int)[source]#

Remove given condition by index.

Parameters

indexint: index of condtion to remove

run(train_dataset: VisionData, test_dataset: VisionData, random_state: int = 42, with_display: bool = True) → CheckResult[source]#

Run check.

Parameters

train_datasetVisionData: VisionData object, representing data the model was fitted on
test_datasetVisionData: VisionData object, representing data the models predicts on
random_stateint: A seed to set for pseudo-random functions
with_displaybool , default: True: flag that determines if checks will calculate display (redundant in some checks).

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) → str[source]#: Serialize check instance to JSON string.

update(context: Context, batch: Any, dataset_kind: DatasetKind)[source]#: Update internal check state with given batch for either train or test.

class ModelOnlyCheck[source]#

Parent class for checks that only use a model and no datasets.

Methods

`add_condition`(name, condition_func, **params)	Add new condition function to the check.
`clean_conditions`()	Remove all conditions from this check instance.
`compute`(context)	Compute final check result.
`conditions_decision`(result)	Run conditions on given result.
`config`([include_version, include_defaults])	Return check configuration (conditions' configuration not yet supported).
`context_type`	alias of `Context`
`from_config`(conf[, version_unmatch])	Return check object from a CheckConfig object.
`from_json`(conf[, version_unmatch])	Deserialize check instance from JSON string.
`initialize_run`(context)	Initialize run before starting updating on batches.
`metadata`([with_doc_link])	Return check metadata.
`name`()	Name of class in split camel case.
`params`([show_defaults])	Return parameters to show when printing the check.
`remove_condition`(index)	Remove given condition by index.
`run`(model[, random_state, with_display])	Run check.
`to_json`([indent, include_version, ...])	Serialize check instance to JSON string.

__init__(**kwargs)[source]#

add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#

Add new condition function to the check.

Parameters

namestr: Name of the condition. should explain the condition action and parameters
condition_funcCallable[[Any], Union[List[ConditionResult], bool]]: Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
paramsdict: Additional parameters to pass when calling the condition function.

clean_conditions()[source]#: Remove all conditions from this check instance.

compute(context: Context) → CheckResult[source]#: Compute final check result.

conditions_decision(result: CheckResult) → List[ConditionResult][source]#: Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) → CheckConfig[source]#

Return check configuration (conditions’ configuration not yet supported).

Returns

CheckConfig: includes the checks class name, params, and module name.

context_type[source]#: alias of Context

classmethod from_config(conf: CheckConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#

Return check object from a CheckConfig object.

Parameters

confDict[Any, Any]

Returns

BaseCheck: the check class object from given config

classmethod from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#: Deserialize check instance from JSON string.

initialize_run(context: Context)[source]#: Initialize run before starting updating on batches. Optional.

metadata(with_doc_link: bool = False) → CheckMetadata[source]#

Return check metadata.

Parameters

with_doc_linkbool, default False: whethere to include doc link in summary or not

Returns

Dict[str, Any]

classmethod name() → str[source]#: Name of class in split camel case.

params(show_defaults: bool = False) → Dict[source]#: Return parameters to show when printing the check.

remove_condition(index: int)[source]#

Remove given condition by index.

Parameters

indexint: index of condtion to remove

run(model, random_state: int = 42, with_display: bool = True) → CheckResult[source]#

Run check.

Parameters

model: Model to run the check on
random_stateint: A seed to set for pseudo-random functions
with_displaybool , default: True: flag that determines if checks will calculate display (redundant in some checks).

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) → str[source]#: Serialize check instance to JSON string.

class Suite[source]#

Tabular suite to run checks of types: TrainTestCheck, SingleDatasetCheck, ModelOnlyCheck.

Methods

`add`(check)	Add a check or a suite to current suite.
`config`()	Return suite configuration (checks' conditions' configuration not yet supported).
`from_config`(conf[, version_unmatch])	Return suite object from a CheckConfig object.
`from_json`(conf[, version_unmatch])	Deserialize suite instance from JSON string.
`remove`(index)	Remove a check by given index.
`run`([train_dataset, test_dataset, ...])	Run all checks.
`supported_checks`()	Return tuple of supported check types of this suite.
`to_json`([indent])	Serialize suite instance to JSON string.

__init__(name: str, *checks: Union[BaseCheck, BaseSuite])[source]#

add(check: Union[BaseCheck, BaseSuite])[source]#

Add a check or a suite to current suite.

Parameters

checkBaseCheck: A check or suite to add.

config() → SuiteConfig[source]#

Return suite configuration (checks’ conditions’ configuration not yet supported).

Returns

SuiteConfig: includes the suite name, and list of check configs.

classmethod from_config(conf: SuiteConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#

Return suite object from a CheckConfig object.

Parameters

confSuiteConfig: the SuiteConfig object

Returns

BaseSuite: the suite class object from given config

from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self[source]#: Deserialize suite instance from JSON string.

remove(index: int)[source]#

Remove a check by given index.

Parameters

indexint: Index of check to remove.

run(train_dataset: Optional[VisionData] = None, test_dataset: Optional[VisionData] = None, random_state: int = 42, with_display: bool = True, max_samples: Optional[int] = None, run_single_dataset: Optional[str] = None) → SuiteResult[source]#

Run all checks.

Parameters

train_datasetOptional[VisionData] , default: None: VisionData object, representing data the model was fitted on
test_datasetOptional[VisionData] , default: None: VisionData object, representing data the models predicts on
random_stateint: A seed to set for pseudo-random functions
with_displaybool , default: True: flag that determines if checks will calculate display (redundant in some checks).
max_samplesOptional[int] , default: None: Each check will run on a number of samples which is the minimum between the n_samples parameter of the check and this parameter. If this argument is None then the number of samples for each check will be determined by the n_samples argument.
run_single_dataset: Optional[str], default None: ‘Train’, ‘Test’ , or None to run on both train and test.

Returns

SuiteResult: All results by all initialized checks

classmethod supported_checks() → Tuple[source]#: Return tuple of supported check types of this suite.

to_json(indent: int = 3) → str[source]#: Serialize suite instance to JSON string.

Functions

classification_dataset_from_directory(root: str, batch_size: int = 32, num_workers: int = 0, shuffle: bool = True, pin_memory: bool = True, object_type: ~typing.Literal[<class 'deepchecks.vision.vision_data.vision_data.VisionData'>, <class 'torch.utils.data.dataloader.DataLoader'>] = 'DataLoader', **kwargs) → Union[Tuple[Union[DataLoader, VisionData]], DataLoader, VisionData][source]#

Load a simple classification dataset.

The function expects that the data within the root folder to be structured one of the following ways:

root/

class1/
image1.jpeg

root/

train/

class1/
image1.jpeg

test/

class1/
image1.jpeg

Parameters

rootstr: path to the data
batch_sizeint, default: 32: Batch size for the dataloader.
num_workersint, default: 0: Number of workers for the dataloader.
shufflebool, default: True: Whether to shuffle the dataset.
pin_memorybool, default: True: If True, the data loader will copy Tensors into CUDA pinned memory before returning them.
object_typeLiteral[‘Dataset’, ‘DataLoader’], default: ‘DataLoader’: type of the return value. If ‘Dataset’, deepchecks.vision.VisionData will be returned, otherwise torch.utils.data.DataLoader

Returns

t.Union[t.Tuple[t.Union[DataLoader, vision.ClassificationData]], t.Union[DataLoader, vision.ClassificationData]]: A DataLoader or VisionDataset instance or tuple representing a single dataset or train and test datasets.

ensure_predictions_proba

checks