train_test_validation#

train_test_validation(n_top_show: int = 5, label_properties: Optional[List[Dict[str, Any]]] = None, image_properties: Optional[List[Dict[str, Any]]] = None, sample_size: Optional[int] = None, random_state: Optional[int] = None, **kwargs) → Suite[source]#

Suite for validating correctness of train-test split, including distribution, integrity and leakage checks.

List of Checks:

List of Checks#
Check Example	API Reference
New Labels	`NewLabels`
Similar Image Leakage	`SimilarImageLeakage`
Heatmap Comparison	`HeatmapComparison`
Train Test Label Drift	`TrainTestLabelDrift`
Image Property Drift	`ImagePropertyDrift`
Image Dataset Drift	`ImageDatasetDrift`
Property Label Correlation Change	`PropertyLabelCorrelationChange`

Parameters

n_top_show: int, default: 5

Number of images to show for checks that show images.

label_propertiesList[Dict[str, Any]], default: None

List of properties. Replaces the default deepchecks properties. Each property is a dictionary with keys ‘name’ (str), ‘method’ (Callable) and ‘output_type’ (str), representing attributes of said method. ‘output_type’ must be one of: - ‘numeric’ - for continuous ordinal outputs. - ‘categorical’ - for discrete, non-ordinal outputs. These can still be numbers,

but these numbers do not have inherent value.

For more on image / label properties, see the vision_properties_guide. - ‘class_id’ - for properties that return the class_id. This is used because these

properties are later matched with the VisionData.label_map, if one was given.

image_propertiesList[Dict[str, Any]], default: None

but these numbers do not have inherent value.

For more on image / label properties, see the vision_properties_guide.

sample_sizeint , default: None

Number of samples to use for checks that sample data. If none, using the default sample_size per check.

random_state: int, default: None

Random seed for all checks.

**kwargsdict

additional arguments to pass to the checks.

Returns

Suite: A Suite for validating correctness of train-test split, including distribution, integrity and leakage checks.

See also

Classification Model Validation Tutorial
Object Detection Tutorial

Examples

>>> from deepchecks.vision.suites import train_test_validation
>>> suite = train_test_validation(n_top_show=3, sample_size=100)
>>> result = suite.run()
>>> result.show()

run(self, train_dataset: Optional[VisionData] = None, test_dataset: Optional[VisionData] = None, model: Optional[Module] = None, scorers: Optional[Mapping[str, Metric]] = None, scorers_per_class: Optional[Mapping[str, Metric]] = None, device: Optional[Union[str, device]] = None, random_state: int = 42, with_display: bool = True, n_samples: Optional[int] = None, train_predictions: Optional[Dict[int, Union[Sequence[Tensor], Tensor]]] = None, test_predictions: Optional[Dict[int, Union[Sequence[Tensor], Tensor]]] = None, model_name: str = '') → SuiteResult#

Run all checks.

Parameters

train_dataset: Optional[VisionData] , default None: object, representing data an estimator was fitted on
test_datasetOptional[VisionData] , default None: object, representing data an estimator predicts on
modelnn.Module , default None: A scikit-learn-compatible fitted estimator instance
model_name: str , default: ‘’: The name of the model
scorersOptional[Mapping[str, Metric]] , default: None: dict of scorers names to a Metric
scorers_per_classOptional[Mapping[str, Metric]] , default: None: dict of scorers for classification without averaging of the classes. See <a href= “https://scikit-learn.org/stable/modules/model_evaluation.html#from-binary-to-multiclass-and-multilabel”> scikit-learn docs</a>
deviceUnion[str, torch.device], default: ‘cpu’: processing unit for use
random_stateint: A seed to set for pseudo-random functions
n_samplesOptional[int], default: None: number of samples
with_displaybool , default: True: flag that determines if checks will calculate display (redundant in some checks).
train_predictions: Optional[Dict[int, Union[Sequence[torch.Tensor], torch.Tensor]]] , default None: Dictionary of the model prediction over the train dataset (keys are the indexes).
test_predictions: Optional[Dict[int, Union[Sequence[torch.Tensor], torch.Tensor]]] , default None: Dictionary of the model prediction over the test dataset (keys are the indexes).

Returns

SuiteResult: All results by all initialized checks

data_integrity

model_evaluation