deepchecks.vision#
Package for vision functionality.
Modules
Module importing all vision checks. |
|
Module contains all prebuilt vision suites. |
|
Module containing datasets and models for vision tasks. |
|
Package for vision utilities. |
|
Module containing metrics utils. |
|
Package for vision data class and utilities. |
|
Module for vision base checks. |
|
Module for base vision context. |
|
This file changes default 'ignore' action of DeprecationWarnings for specific deprecation messages. |
|
Module containing metrics implemented by Deepchecks. |
|
Module for base vision abstractions. |
Classes
- class VisionData[source]#
VisionData is the base data object of deepcheck vision used as input to checks and suites.
- Parameters
- batch_loader
A batch_loader which load a batch of data in an iterative manner. Batch loader batch output must be a dictionary in BatchOutputFormat format. The batch loader must provide SHUFFLED batches.
- task_typestr
The task type of the data. can be one of the following: ‘classification’, ‘semantic_segmentation’, ‘object_detection’, ‘other’. For ‘other’, only image related checks (such as ImagePropertyOutliers) will be run.
- label_mapDict[int, str], optional
A dictionary mapping class ids to their names.
- dataset_name: str, optional
Name of the dataset to use in the displays instead of “Train” or “Test”.
- reshuffle_data: bool, default=True
If True we will attempt to shuffle the batch loader. Only set this to False if the data is already shuffled.
- Attributes
batch_loader
Return the batch loader used be the vision data.
has_additional_data
Return True if the batch loader contains additional_data.
has_embeddings
Return True if the batch loader contains embeddings.
has_image_identifiers
Return True if the batch loader contains image identifiers.
has_images
Return True if the batch loader contains images.
has_labels
Return True if the batch loader contains labels.
has_predictions
Return True if the batch loader contains predictions.
num_classes
Return a number of possible classes based on model proba, label map or observed classes.
number_of_images_cached
Return True if the number of images processed and whose statistics were cached.
task_type
Return True if the batch loader contains labels.
Methods
copy
([reshuffle_data, batch_loader])Create new copy of the vision data object with clean cache.
get_cache
([use_class_names])Return a dictionary of stored cache.
get_observed_classes
([use_class_names])Return a dictionary of observed classes either as class ids or as the class names.
head
([num_images_to_display, show_in_window])Show data from a single batch of this VisionData.
Initialize the cache.
update_cache
(batch_size, numpy_labels, ...)Update cache based on newly arrived batch.
validate
()Validate the VisionData functionalities implemented by the user and set which formatters were implemented.
- __init__(batch_loader, task_type: typing_extensions.Literal[classification, object_detection, semantic_segmentation, other], label_map: Optional[Dict[int, str]] = None, dataset_name: Optional[str] = None, reshuffle_data: bool = True)[source]#
- property batch_loader#
Return the batch loader used be the vision data.
- copy(reshuffle_data: bool = False, batch_loader=None) VD [source]#
Create new copy of the vision data object with clean cache.
- Parameters
- reshuffle_data: bool, default=False
If True and the batch loader is of known type that can be shuffled, it will be shuffled.
- batch_loader:
If not None, the batch loader of the new object will be set to this value.
- Returns
- VisionData
A copy of the vision data object with clean cache.
- get_cache(use_class_names: bool = True) Dict[str, Any] [source]#
Return a dictionary of stored cache.
- get_observed_classes(use_class_names: bool = True) List[str] [source]#
Return a dictionary of observed classes either as class ids or as the class names.
- head(num_images_to_display: int = 5, show_in_window: bool = False)[source]#
Show data from a single batch of this VisionData. Works only inside a notebook.
- Parameters
- num_images_to_display: int, default = 5
Number of images to show. Does not show more images than the size of single batch
- show_in_window: bool, default = False
Whether to open the head display in a new python window. requires pyqt5, pyqtwebengine libraries.
- property num_classes: int#
Return a number of possible classes based on model proba, label map or observed classes.
- property number_of_images_cached: int#
Return True if the number of images processed and whose statistics were cached.
- class BatchOutputFormat[source]#
Batch output format required by deepchecks.
Methods
clear
()copy
()fromkeys
(iterable[, value])Create a new dictionary with keys from iterable and values set to value.
get
(key[, default])Return the value for key if key is in the dictionary, else default.
items
()keys
()pop
(key[, default])If key is not found, default is returned if given, otherwise KeyError is raised
popitem
(/)Remove and return a (key, value) pair as a 2-tuple.
setdefault
(key[, default])Insert key with a value of default if key is not in the dictionary.
update
([E, ]**F)If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
values
()- __init__(*args, **kwargs)#
- __new__(*args, **kwargs)#
- clear() None. Remove all items from D. #
- copy() a shallow copy of D #
- fromkeys(iterable, value=None, /)#
Create a new dictionary with keys from iterable and values set to value.
- get(key, default=None, /)#
Return the value for key if key is in the dictionary, else default.
- items() a set-like object providing a view on D's items #
- keys() a set-like object providing a view on D's keys #
- pop(key, default=<unrepresentable>, /)#
If key is not found, default is returned if given, otherwise KeyError is raised
- popitem(/)#
Remove and return a (key, value) pair as a 2-tuple.
Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.
- setdefault(key, default=None, /)#
Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
- update([E, ]**F) None. Update D from dict/iterable E and F. #
If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
- values() an object providing a view on D's values #
- class SingleDatasetCheck[source]#
Parent class for checks that only use one dataset.
Methods
add_condition
(name, condition_func, **params)Add new condition function to the check.
Remove all conditions from this check instance.
compute
(context, dataset_kind)Compute final check result based on accumulated internal state.
conditions_decision
(result)Run conditions on given result.
config
([include_version, include_defaults])Return check configuration (conditions' configuration not yet supported).
alias of
Context
from_config
(conf[, version_unmatch])Return check object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize check instance from JSON string.
initialize_run
(context, dataset_kind)Initialize run before starting updating on batches.
metadata
([with_doc_link])Return check metadata.
name
()Name of class in split camel case.
params
([show_defaults])Return parameters to show when printing the check.
remove_condition
(index)Remove given condition by index.
run
(dataset[, random_state, with_display])Run check.
to_json
([indent, include_version, ...])Serialize check instance to JSON string.
update
(context, batch, dataset_kind)Update internal check state with given batch.
- add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#
Add new condition function to the check.
- Parameters
- namestr
Name of the condition. should explain the condition action and parameters
- condition_funcCallable[[Any], Union[List[ConditionResult], bool]]
Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
- paramsdict
Additional parameters to pass when calling the condition function.
- compute(context: Context, dataset_kind: DatasetKind) CheckResult [source]#
Compute final check result based on accumulated internal state.
- conditions_decision(result: CheckResult) List[ConditionResult] [source]#
Run conditions on given result.
- config(include_version: bool = True, include_defaults: bool = True) CheckConfig [source]#
Return check configuration (conditions’ configuration not yet supported).
- Returns
- CheckConfig
includes the checks class name, params, and module name.
- classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return check object from a CheckConfig object.
- Parameters
- confDict[Any, Any]
- Returns
- BaseCheck
the check class object from given config
- classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize check instance from JSON string.
- initialize_run(context: Context, dataset_kind: DatasetKind)[source]#
Initialize run before starting updating on batches. Optional.
- metadata(with_doc_link: bool = False) CheckMetadata [source]#
Return check metadata.
- Parameters
- with_doc_linkbool, default False
whethere to include doc link in summary or not
- Returns
- Dict[str, Any]
- params(show_defaults: bool = False) Dict [source]#
Return parameters to show when printing the check.
- remove_condition(index: int)[source]#
Remove given condition by index.
- Parameters
- indexint
index of condtion to remove
- run(dataset: VisionData, random_state: int = 42, with_display: bool = True) CheckResult [source]#
Run check.
- Parameters
- dataset: VisionData
VisionData object to process
- random_stateint
A seed to set for pseudo-random functions
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) str [source]#
Serialize check instance to JSON string.
- update(context: Context, batch: Any, dataset_kind: DatasetKind)[source]#
Update internal check state with given batch.
- class TrainTestCheck[source]#
Parent class for checks that compare two datasets.
The class checks train dataset and test dataset for model training and test.
Methods
add_condition
(name, condition_func, **params)Add new condition function to the check.
Remove all conditions from this check instance.
compute
(context)Compute final check result based on accumulated internal state.
conditions_decision
(result)Run conditions on given result.
config
([include_version, include_defaults])Return check configuration (conditions' configuration not yet supported).
alias of
Context
from_config
(conf[, version_unmatch])Return check object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize check instance from JSON string.
initialize_run
(context)Initialize run before starting updating on batches.
metadata
([with_doc_link])Return check metadata.
name
()Name of class in split camel case.
params
([show_defaults])Return parameters to show when printing the check.
remove_condition
(index)Remove given condition by index.
run
(train_dataset, test_dataset[, ...])Run check.
to_json
([indent, include_version, ...])Serialize check instance to JSON string.
update
(context, batch, dataset_kind)Update internal check state with given batch for either train or test.
- add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#
Add new condition function to the check.
- Parameters
- namestr
Name of the condition. should explain the condition action and parameters
- condition_funcCallable[[Any], Union[List[ConditionResult], bool]]
Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
- paramsdict
Additional parameters to pass when calling the condition function.
- compute(context: Context) CheckResult [source]#
Compute final check result based on accumulated internal state.
- conditions_decision(result: CheckResult) List[ConditionResult] [source]#
Run conditions on given result.
- config(include_version: bool = True, include_defaults: bool = True) CheckConfig [source]#
Return check configuration (conditions’ configuration not yet supported).
- Returns
- CheckConfig
includes the checks class name, params, and module name.
- classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return check object from a CheckConfig object.
- Parameters
- confDict[Any, Any]
- Returns
- BaseCheck
the check class object from given config
- classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize check instance from JSON string.
- initialize_run(context: Context)[source]#
Initialize run before starting updating on batches. Optional.
- metadata(with_doc_link: bool = False) CheckMetadata [source]#
Return check metadata.
- Parameters
- with_doc_linkbool, default False
whethere to include doc link in summary or not
- Returns
- Dict[str, Any]
- params(show_defaults: bool = False) Dict [source]#
Return parameters to show when printing the check.
- remove_condition(index: int)[source]#
Remove given condition by index.
- Parameters
- indexint
index of condtion to remove
- run(train_dataset: VisionData, test_dataset: VisionData, random_state: int = 42, with_display: bool = True) CheckResult [source]#
Run check.
- Parameters
- train_datasetVisionData
VisionData object, representing data the model was fitted on
- test_datasetVisionData
VisionData object, representing data the models predicts on
- random_stateint
A seed to set for pseudo-random functions
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) str [source]#
Serialize check instance to JSON string.
- update(context: Context, batch: Any, dataset_kind: DatasetKind)[source]#
Update internal check state with given batch for either train or test.
- class ModelOnlyCheck[source]#
Parent class for checks that only use a model and no datasets.
Methods
add_condition
(name, condition_func, **params)Add new condition function to the check.
Remove all conditions from this check instance.
compute
(context)Compute final check result.
conditions_decision
(result)Run conditions on given result.
config
([include_version, include_defaults])Return check configuration (conditions' configuration not yet supported).
alias of
Context
from_config
(conf[, version_unmatch])Return check object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize check instance from JSON string.
initialize_run
(context)Initialize run before starting updating on batches.
metadata
([with_doc_link])Return check metadata.
name
()Name of class in split camel case.
params
([show_defaults])Return parameters to show when printing the check.
remove_condition
(index)Remove given condition by index.
run
(model[, random_state, with_display])Run check.
to_json
([indent, include_version, ...])Serialize check instance to JSON string.
- add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#
Add new condition function to the check.
- Parameters
- namestr
Name of the condition. should explain the condition action and parameters
- condition_funcCallable[[Any], Union[List[ConditionResult], bool]]
Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
- paramsdict
Additional parameters to pass when calling the condition function.
- compute(context: Context) CheckResult [source]#
Compute final check result.
- conditions_decision(result: CheckResult) List[ConditionResult] [source]#
Run conditions on given result.
- config(include_version: bool = True, include_defaults: bool = True) CheckConfig [source]#
Return check configuration (conditions’ configuration not yet supported).
- Returns
- CheckConfig
includes the checks class name, params, and module name.
- classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return check object from a CheckConfig object.
- Parameters
- confDict[Any, Any]
- Returns
- BaseCheck
the check class object from given config
- classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize check instance from JSON string.
- initialize_run(context: Context)[source]#
Initialize run before starting updating on batches. Optional.
- metadata(with_doc_link: bool = False) CheckMetadata [source]#
Return check metadata.
- Parameters
- with_doc_linkbool, default False
whethere to include doc link in summary or not
- Returns
- Dict[str, Any]
- params(show_defaults: bool = False) Dict [source]#
Return parameters to show when printing the check.
- remove_condition(index: int)[source]#
Remove given condition by index.
- Parameters
- indexint
index of condtion to remove
- run(model, random_state: int = 42, with_display: bool = True) CheckResult [source]#
Run check.
- Parameters
- model
Model to run the check on
- random_stateint
A seed to set for pseudo-random functions
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- class Suite[source]#
Tabular suite to run checks of types: TrainTestCheck, SingleDatasetCheck, ModelOnlyCheck.
Methods
add
(check)Add a check or a suite to current suite.
config
()Return suite configuration (checks' conditions' configuration not yet supported).
from_config
(conf[, version_unmatch])Return suite object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize suite instance from JSON string.
remove
(index)Remove a check by given index.
run
([train_dataset, test_dataset, ...])Run all checks.
Return tuple of supported check types of this suite.
to_json
([indent])Serialize suite instance to JSON string.
- add(check: Union[BaseCheck, BaseSuite])[source]#
Add a check or a suite to current suite.
- Parameters
- checkBaseCheck
A check or suite to add.
- config() SuiteConfig [source]#
Return suite configuration (checks’ conditions’ configuration not yet supported).
- Returns
- SuiteConfig
includes the suite name, and list of check configs.
- classmethod from_config(conf: ~deepchecks.core.suite.SuiteConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return suite object from a CheckConfig object.
- Parameters
- confSuiteConfig
the SuiteConfig object
- Returns
- BaseSuite
the suite class object from given config
- from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize suite instance from JSON string.
- remove(index: int)[source]#
Remove a check by given index.
- Parameters
- indexint
Index of check to remove.
- run(train_dataset: Optional[VisionData] = None, test_dataset: Optional[VisionData] = None, random_state: int = 42, with_display: bool = True, max_samples: Optional[int] = None, run_single_dataset: Optional[str] = None) SuiteResult [source]#
Run all checks.
- Parameters
- train_datasetOptional[VisionData] , default: None
VisionData object, representing data the model was fitted on
- test_datasetOptional[VisionData] , default: None
VisionData object, representing data the models predicts on
- random_stateint
A seed to set for pseudo-random functions
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- max_samplesOptional[int] , default: None
Each check will run on a number of samples which is the minimum between the n_samples parameter of the check and this parameter. If this argument is None then the number of samples for each check will be determined by the n_samples argument.
- run_single_dataset: Optional[str], default None
‘Train’, ‘Test’ , or None to run on both train and test.
- Returns
- SuiteResult
All results by all initialized checks
Functions
- classification_dataset_from_directory(root: str, batch_size: int = 32, num_workers: int = 0, shuffle: bool = True, pin_memory: bool = True, object_type: typing_extensions.Literal[VisionData, DataLoader] = 'DataLoader', **kwargs) Union[Tuple[Union[DataLoader, VisionData]], DataLoader, VisionData] [source]#
Load a simple classification dataset.
The function expects that the data within the root folder to be structured one of the following ways:
- root/
- class1/
image1.jpeg
- root/
- train/
- class1/
image1.jpeg
- test/
- class1/
image1.jpeg
- Parameters
- rootstr
path to the data
- batch_sizeint, default: 32
Batch size for the dataloader.
- num_workersint, default: 0
Number of workers for the dataloader.
- shufflebool, default: True
Whether to shuffle the dataset.
- pin_memorybool, default: True
If
True
, the data loader will copy Tensors into CUDA pinned memory before returning them.- object_typeLiteral[‘Dataset’, ‘DataLoader’], default: ‘DataLoader’
type of the return value. If ‘Dataset’,
deepchecks.vision.VisionData
will be returned, otherwisetorch.utils.data.DataLoader
- Returns
- t.Union[t.Tuple[t.Union[DataLoader, vision.ClassificationData]], t.Union[DataLoader, vision.ClassificationData]]
A DataLoader or VisionDataset instance or tuple representing a single dataset or train and test datasets.