deepchecks.nlp

Package for nlp functionality.

Modules

checks	Module importing all nlp checks.
suites	Module contains all prebuilt nlp suites.
datasets	Module for working with pre-built datasets.
metric_utils	Module containing metrics utils for nlp tasks.
utils	Utils package for nlp functionality.
text_data	The dataset module containing the tabular Dataset class and its functions.
base_checks	Module for nlp base checks.
context	Module for base nlp context.
suite	Module for base tabular abstractions.

Classes

class TextData

TextData wraps together the raw text data and the labels for the nlp task.

The TextData class contains metadata and methods intended for easily accessing metadata relevant for the training or validating of ML models.

Parameters

raw_textt.Sequence[str], default: None

The raw text data, a sequence of strings representing the raw text of each sample. If not given, tokenized_text must be given, and raw_text will be created from it by joining the tokens with spaces.

tokenized_textt.Sequence[t.Sequence[str]], default: None

The tokenized text data, a sequence of sequences of strings representing the tokenized text of each sample. Only relevant for task_type ‘token_classification’. If not given, raw_text must be given, and tokenized_text will be created from it by splitting the text by spaces.

labelt.Optional[TTextLabel], default: None

The label for the text data. Can be either a text_classification label or a token_classification label. If None, the label is not set.

• text_classification label - For text classification the accepted label format differs between multilabel and single label cases. For single label data, the label should be passed as a sequence of labels, with one entry per sample that can be either a string or an integer. For multilabel data, the label should be passed as a sequence of sequences, with the sequence for each sample being a binary vector, representing the presence of the i-th label in that sample.

• token_classification label - For token classification the accepted label format is the IOB format or similar to it. The Label must be a sequence of sequences of strings or integers, with each sequence corresponding to a sample in the tokenized text, and exactly the length of the corresponding tokenized text.

task_typestr, default: None

The task type for the text data. Can be either ‘text_classification’ or ‘token_classification’. Must be set if label is provided.

dataset_namet.Optional[str] , default: None

The name of the dataset. If None, the dataset name will be defined when running it within a check.

indext.Optional[t.Sequence[int]] , default: None

The index of the samples. If None, the index is set to np.arange(len(raw_text)).

metadatat.Optional[pd.DataFrame] , default: None

Metadata for the samples. If None, no metadata is set. If a DataFrame is given, it must contain the same number of samples as the raw_text and identical index.

propertiest.Optional[Union[pd.DataFrame, str]] , default: None

The text properties for the samples. If None, no properties are set. If ‘auto’, the properties are calculated using the default properties. If a DataFrame is given, it must contain the properties for each sample as the raw text and identical index.

devicestr, default: None

The device to use to calculate the text properties.

Attributes

is_multilabel

Return True if label is multilabel.

label

Return the label defined in the dataset.

metadata

Return the metadata of for the dataset.

metadata_types

Return the metadata types of for the dataset.

n_samples

Return number of samples in the dataset.

name

properties

Return the properties of the dataset.

properties_types

Return the property types of the dataset.

task_type

Return the task type.

text

Return sequence of raw text samples.

tokenized_text

Return sequence of tokenized text samples.

Methods

calculate_default_properties([...])	Calculate the default properties of the dataset.
cast_to_dataset(obj)	Verify Dataset or transform to Dataset.
copy([rows_to_use])	Create a copy of this Dataset with new data.
datasets_share_task_type(*datasets)	Verify that all provided datasets share same label name and task types.
get_raw_sample(index)	Get the raw text of a sample.
get_tokenized_sample(index)	Get the tokenized text of a sample.
has_label()	Return True if label was set.
head([n_samples])	Return a copy of the dataset as a pandas Dataframe with the first n_samples samples.
is_sampled(n_samples)	Return True if the dataset number of samples will decrease when sampled with n_samples samples.
len_when_sampled(n_samples)	Return number of samples in the sampled dataframe this dataset is sampled with n_samples samples.
reindex(index)	Reindex the TextData with a new index.
sample(n_samples[, replace, random_state, ...])	Create a copy of the dataset object, with the internal data being a sample of the original data.
set_metadata(metadata[, metadata_types])	Set the metadata of the dataset.
set_properties(properties[, properties_types])	Set the properties of the dataset.

__init__(raw_text: Optional[Sequence[str]] = None, tokenized_text: Optional[Sequence[Sequence[str]]] = None, label: Optional[Union[Sequence[Union[Tuple[int, str], Tuple[Tuple[int, str]]]], Sequence[Sequence[Union[str, int]]]]] = None, task_type: Optional[str] = None, dataset_name: Optional[str] = None, index: Optional[Sequence[Any]] = None, metadata: Optional[DataFrame] = None, properties: Optional[Union[DataFrame, str]] = None, device: Optional[str] = None)

calculate_default_properties(include_properties: Optional[List[str]] = None, ignore_properties: Optional[List[str]] = None, include_long_calculation_properties: Optional[bool] = False, device: Optional[str] = None)

Calculate the default properties of the dataset.

Parameters

include_propertiesList[str], default None

The properties to calculate. If None, all default properties will be calculated. Cannot be used together with ignore_properties parameter.

ignore_propertiesList[str], default None

The properties to ignore. If None, no properties will be ignored. Cannot be used together with properties parameter.

include_long_calculation_propertiesbool, default False

Whether to include properties that may take a long time to calculate. If False, these properties will be ignored.

deviceint, default None

The device to use for the calculation. If None, the default device will be used.

classmethod cast_to_dataset(obj: Any) → TextData

Verify Dataset or transform to Dataset.

Function verifies that provided value is a non-empty instance of Dataset, otherwise raises an exception, but if the ‘cast’ flag is set to True it will also try to transform provided value to the Dataset instance.

Parameters

obj

value to verify

Raises

DeepchecksValueError

if the provided value is not a TextData instance; if the provided value cannot be transformed into Dataset instance;

copy(rows_to_use: Optional[Sequence[Any]] = None) → TDataset

Create a copy of this Dataset with new data.

classmethod datasets_share_task_type(*datasets: TextData) → bool

Verify that all provided datasets share same label name and task types.

Parameters

datasetsList[TextData]

list of TextData to validate

Returns

bool

True if all TextData share same label names and task types, otherwise False

Raises

AssertionError

‘datasets’ parameter is not a list; ‘datasets’ contains less than one dataset;

get_raw_sample(index: Any) → str

Get the raw text of a sample.

Parameters

indexint

Index of sample to get.

Returns

——-

str

Raw text of sample.

get_tokenized_sample(index: Any) → List[str]

Get the tokenized text of a sample.

Parameters

indexint

Index of sample to get.

Returns

——-

List[str]

Tokenized text of sample.

has_label() → bool

Return True if label was set.

Returns

bool

True if label was set.

head(n_samples: int = 5) → DataFrame

Return a copy of the dataset as a pandas Dataframe with the first n_samples samples.

property is_multilabel: bool

Return True if label is multilabel.

Returns

bool

True if label is multilabel.

is_sampled(n_samples: Optional[int])

Return True if the dataset number of samples will decrease when sampled with n_samples samples.

property label: Union[Sequence[Union[Tuple[int, str], Tuple[Tuple[int, str]]]], Sequence[Sequence[Union[str, int]]]]

Return the label defined in the dataset.

Returns

TTextLabel

len_when_sampled(n_samples: Optional[int])

Return number of samples in the sampled dataframe this dataset is sampled with n_samples samples.

property metadata: DataFrame

Return the metadata of for the dataset.

property metadata_types: Dict[str, str]

Return the metadata types of for the dataset.

property n_samples: int

Return number of samples in the dataset.

property properties: DataFrame

Return the properties of the dataset.

property properties_types: Dict[str, str]

Return the property types of the dataset.

reindex(index: Sequence[Any])

Reindex the TextData with a new index.

sample(n_samples: int, replace: bool = False, random_state: Optional[int] = None, drop_na_label: bool = False) → TDataset

Create a copy of the dataset object, with the internal data being a sample of the original data.

Parameters

n_samplesint

Number of samples to draw.

replacebool, default: False

Whether to sample with replacement.

random_statet.Optional[int] , default None

Random state.

drop_na_labelbool, default: False

Whether to take sample only from rows with exiting label.

Returns

Dataset

instance of the Dataset with sampled internal dataframe.

set_metadata(metadata: DataFrame, metadata_types: Optional[Dict[str, str]] = None)

Set the metadata of the dataset.

set_properties(properties: DataFrame, properties_types: Optional[Dict[str, str]] = None)

Set the properties of the dataset.

property task_type: Optional[deepchecks.nlp.task_type.TaskType]

Return the task type.

Returns

t.Optional[TaskType]

Task type

property text: Sequence[str]

Return sequence of raw text samples.

Returns

t.Sequence[str]

Sequence of raw text samples.

property tokenized_text: Sequence[Sequence[str]]

Return sequence of tokenized text samples.

Returns

t.Sequence[t.Sequence[str]]

Sequence of tokenized text samples.

class Context

Contains all the data + properties the user has passed to a check/suite, and validates it seamlessly.

Parameters

train_datasetUnion[TextData, None] , default: None

TextData object, representing data an estimator was fitted on

test_datasetUnion[TextData, None] , default: None

TextData object, representing data an estimator predicts on

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

train_predUnion[TTextPred, None] , default: None

predictions on train dataset

test_predUnion[TTextPred, None] , default: None

predictions on test dataset

train_probaUnion[TTextProba, None] , default: None

probabilities on train dataset

test_probaUnion[TTextProba, None] , default: None

probabilities on test dataset

model_classesOptional[List] , default: None

list of classes known to the model

random_state: int, default 42

A seed to set for pseudo-random functions , primarily sampling.

Attributes

model

Return model if exists, otherwise raise error.

model_classes

Return ordered list of possible label classes for classification tasks.

model_name

Return the name of the model.

observed_classes

Return the observed classes in both train and test.

task_type

Return task type if model & train_dataset & label exists.

test

Return test if exists, otherwise raise error.

train

Return train if exists, otherwise raise error.

with_display

Return the with_display flag.

Methods

assert_metadata(text_data)	Assert that metadata exists.
assert_properties(text_data)	Assert that properties exists.
assert_task_type(*expected_types)	Assert task_type matching given types.
finalize_check_result(check_result, check[, ...])	Run final processing on a check result which includes validation, conditions processing and sampling footnote.
get_data_by_kind(kind)	Return the relevant Dataset by given kind.
get_scorers([scorers, use_avg_defaults])	Return initialized & validated scorers if provided or default scorers otherwise.
get_single_scorer([scorer, use_avg_defaults])	Return initialized & validated scorer if provided or a default scorer otherwise.
have_test()	Return whether there is test_dataset dataset defined.
infer_task_type(train_dataset, test_dataset)	Infer the task type.

__init__(train_dataset: Optional[TextData] = None, test_dataset: Optional[TextData] = None, with_display: bool = True, train_pred: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, test_pred: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, train_proba: Optional[Sequence[Sequence[float]]] = None, test_proba: Optional[Sequence[Sequence[float]]] = None, model_classes: Optional[List] = None, random_state: int = 42)

static assert_metadata(text_data)

Assert that metadata exists.

static assert_properties(text_data)

Assert that properties exists.

assert_task_type(*expected_types: TaskType)

Assert task_type matching given types.

If task_type is defined, validate it and raise error if needed, else returns True. If task_type is not defined, return False.

finalize_check_result(check_result, check, dataset_kind: Optional[DatasetKind] = None)

Run final processing on a check result which includes validation, conditions processing and sampling footnote.

get_data_by_kind(kind: DatasetKind)

Return the relevant Dataset by given kind.

get_scorers(scorers: Optional[Union[Mapping[str, Union[str, Callable]], List[str]]] = None, use_avg_defaults=True) → List[DeepcheckScorer]

Return initialized & validated scorers if provided or default scorers otherwise.

Parameters

scorersUnion[List[str], Dict[str, Union[str, Callable]]], default: None

List of scorers to use. If None, use default scorers. Scorers can be supplied as a list of scorer names or as a dictionary of names and functions.

use_avg_defaultsbool, default: True

If no scorers were provided, for classification, determines whether to use default scorers that return an averaged metric, or default scorers that return a metric per class.

Returns

——-

List[DeepcheckScorer]

A list of initialized & validated scorers.

get_single_scorer(scorer: Optional[Mapping[str, Union[str, Callable]]] = None, use_avg_defaults=True) → DeepcheckScorer

Return initialized & validated scorer if provided or a default scorer otherwise.

Parameters

scorerUnion[List[str], Dict[str, Union[str, Callable]]], default: None

List of scorers to use. If None, use default scorers. Scorers can be supplied as a list of scorer names or as a dictionary of names and functions.

use_avg_defaultsbool, default True

If no scorers were provided, for classification, determines whether to use default scorers that return an averaged metric, or default scorers that return a metric per class.

Returns

——-

List[DeepcheckScorer]

An initialized & validated scorer.

have_test()

Return whether there is test_dataset dataset defined.

static infer_task_type(train_dataset: TextData, test_dataset: TextData)

Infer the task type.

property model: deepchecks.nlp.context._DummyModel

Return model if exists, otherwise raise error.

property model_classes: List

Return ordered list of possible label classes for classification tasks.

property model_name: str

Return the name of the model.

property observed_classes: List

Return the observed classes in both train and test.

property task_type: deepchecks.nlp.task_type.TaskType

Return task type if model & train_dataset & label exists. otherwise, raise error.

property test

Return test if exists, otherwise raise error.

property train

Return train if exists, otherwise raise error.

property with_display: bool

Return the with_display flag.

class SingleDatasetCheck

Parent class for checks that only use one dataset.

Methods

add_condition(name, condition_func, **params)	Add new condition function to the check.
clean_conditions()	Remove all conditions from this check instance.
conditions_decision(result)	Run conditions on given result.
config([include_version, include_defaults])	Return check configuration (conditions' configuration not yet supported).
context_type	alias of Context
from_config(conf[, version_unmatch])	Return check object from a CheckConfig object.
from_json(conf[, version_unmatch])	Deserialize check instance from JSON string.
metadata([with_doc_link])	Return check metadata.
name()	Name of class in split camel case.
params([show_defaults])	Return parameters to show when printing the check.
remove_condition(index)	Remove given condition by index.
run(dataset[, model, with_display, ...])	Run check.
run_logic(context, dataset_kind)	Run check.
to_json([indent, include_version, ...])	Serialize check instance to JSON string.

__init__(**kwargs)

add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)

Add new condition function to the check.

Parameters

namestr

Name of the condition. should explain the condition action and parameters

condition_funcCallable[[Any], Union[List[ConditionResult], bool]]

Function which gets the value of the check and returns object of List[ConditionResult] or boolean.

paramsdict

Additional parameters to pass when calling the condition function.

clean_conditions()

Remove all conditions from this check instance.

conditions_decision(result: CheckResult) → List[ConditionResult]

Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) → CheckConfig

Return check configuration (conditions’ configuration not yet supported).

Returns

CheckConfig

includes the checks class name, params, and module name.

context_type

alias of Context

classmethod from_config(conf: CheckConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self

Return check object from a CheckConfig object.

Parameters

confDict[Any, Any]

Returns

BaseCheck

the check class object from given config

classmethod from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self

Deserialize check instance from JSON string.

metadata(with_doc_link: bool = False) → CheckMetadata

Return check metadata.

Parameters

with_doc_linkbool, default False

whethere to include doc link in summary or not

Returns

Dict[str, Any]

classmethod name() → str

Name of class in split camel case.

params(show_defaults: bool = False) → Dict

Return parameters to show when printing the check.

remove_condition(index: int)

Remove given condition by index.

Parameters

indexint

index of condtion to remove

run(dataset: TextData, model=None, with_display: bool = True, predictions: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, probabilities: Optional[Sequence[Sequence[float]]] = None, model_classes: Optional[List] = None, random_state: int = 42) → CheckResult

Run check.

Parameters

dataset: TextData

Dataset representing data an estimator was fitted on

model: object, default: None

Dummy model object for compatibility with SingleDatasetBaseCheck

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

predictions: Union[TTextPred, None] , default: None

predictions on dataset

probabilities: Union[TTextProba, None] , default: None

probabilities on dataset

model_classes: Optional[List, List[List], default: None

For classification: list of classes known to the model

random_stateint, default 42

A seed to set for pseudo-random functions, primarily sampling.

Notes

The accepted formats for providing model predictions and probabilities are detailed below

Text Classification

Single Class Predictions

• predictions - A sequence of class names or indices with one entry per sample, matching the set of classes present in the labels.

• probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should sum to 1 for each sample.

Multilabel Predictions

• predictions - A sequence of sequences with each element containing a binary vector denoting the presence of the i-th class for the given sample. Each such vector should have one binary indicator per class according to the class (sorted) order. More than one class can be present for each sample.

• probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should range from 0 to 1 for each sample, but are not required to sum to 1.

Token Classification

• predictions - A sequence of sequences, with the inner sequence containing tuples in the following format: (class_name, span_start, span_end, class_probability). span_start and span_end are the start and end character indices of the token within the text, as it was passed to the raw_text argument. Each upper level sequence contains a sequence of tokens for each sample.

• probabilities - No probabilities should be passed for Token Classification tasks. Passing probabilities will result in an error.

Examples

Text Classification

Single Class Predictions

>>> predictions = ['class_1', 'class_1', 'class_2']
>>> probabilities = [[0.2, 0.8], [0.5, 0.5], [0.3, 0.7]]

Multilabel Predictions

>>> predictions = [[0, 0, 1], [0, 1, 1]]
>>> probabilities = [[0.2, 0.3, 0.8], [0.4, 0.9, 0.6]]

Token Classification

>>> predictions = [[('class_1', 0, 2, 0.8), ('class_2', 7, 10, 0.9)], [('class_2', 42, 54, 0.4)], []]

abstract run_logic(context, dataset_kind) → CheckResult

Run check.

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) → str

Serialize check instance to JSON string.

class TrainTestCheck

Parent class for checks that compare two datasets.

The class checks train dataset and test_dataset dataset for model training and test_dataset.

Methods

add_condition(name, condition_func, **params)	Add new condition function to the check.
clean_conditions()	Remove all conditions from this check instance.
conditions_decision(result)	Run conditions on given result.
config([include_version, include_defaults])	Return check configuration (conditions' configuration not yet supported).
context_type	alias of Context
from_config(conf[, version_unmatch])	Return check object from a CheckConfig object.
from_json(conf[, version_unmatch])	Deserialize check instance from JSON string.
metadata([with_doc_link])	Return check metadata.
name()	Name of class in split camel case.
params([show_defaults])	Return parameters to show when printing the check.
remove_condition(index)	Remove given condition by index.
run(train_dataset, test_dataset[, model, ...])	Run check.
run_logic(context)	Run check.
to_json([indent, include_version, ...])	Serialize check instance to JSON string.

__init__(**kwargs)

add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)

Add new condition function to the check.

Parameters

namestr

Name of the condition. should explain the condition action and parameters

condition_funcCallable[[Any], Union[List[ConditionResult], bool]]

Function which gets the value of the check and returns object of List[ConditionResult] or boolean.

paramsdict

Additional parameters to pass when calling the condition function.

clean_conditions()

Remove all conditions from this check instance.

conditions_decision(result: CheckResult) → List[ConditionResult]

Run conditions on given result.

config(include_version: bool = True, include_defaults: bool = True) → CheckConfig

Return check configuration (conditions’ configuration not yet supported).

Returns

CheckConfig

includes the checks class name, params, and module name.

context_type

alias of Context

classmethod from_config(conf: CheckConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self

Return check object from a CheckConfig object.

Parameters

confDict[Any, Any]

Returns

BaseCheck

the check class object from given config

classmethod from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self

Deserialize check instance from JSON string.

metadata(with_doc_link: bool = False) → CheckMetadata

Return check metadata.

Parameters

with_doc_linkbool, default False

whethere to include doc link in summary or not

Returns

Dict[str, Any]

classmethod name() → str

Name of class in split camel case.

params(show_defaults: bool = False) → Dict

Return parameters to show when printing the check.

remove_condition(index: int)

Remove given condition by index.

Parameters

indexint

index of condtion to remove

run(train_dataset: TextData, test_dataset: TextData, model=None, with_display: bool = True, train_predictions: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, test_predictions: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, train_probabilities: Optional[Sequence[Sequence[float]]] = None, test_probabilities: Optional[Sequence[Sequence[float]]] = None, random_state: int = 42) → CheckResult

Run check.

Parameters

train_dataset: Union[TextData, None] , default: None

TextData object, representing data an estimator was fitted on

test_dataset: Union[TextData, None] , default: None

TextData object, representing data an estimator predicts on

model: object, default: None

Dummy model object for compatibility with TrainTestBaseCheck

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

train_predictions: Union[TTextPred, None] , default: None

predictions on train dataset

test_predictions: Union[TTextPred, None] , default: None

predictions on test_dataset dataset

train_probabilities: Union[TTextProba, None] , default: None

probabilities on train dataset

test_probabilities: Union[TTextProba, None] , default: None

probabilities on test_dataset dataset

random_stateint, default 42

A seed to set for pseudo-random functions, primarily sampling.

Notes

The accepted formats for providing model predictions and probabilities are detailed below

Text Classification

Single Class Predictions

• predictions - A sequence of class names or indices with one entry per sample, matching the set of classes present in the labels.

• probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should sum to 1 for each sample.

Multilabel Predictions

• predictions - A sequence of sequences with each element containing a binary vector denoting the presence of the i-th class for the given sample. Each such vector should have one binary indicator per class according to the class (sorted) order. More than one class can be present for each sample.

• probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should range from 0 to 1 for each sample, but are not required to sum to 1.

Token Classification

• predictions - A sequence of sequences, with the inner sequence containing tuples in the following format: (class_name, span_start, span_end, class_probability). span_start and span_end are the start and end character indices of the token within the text, as it was passed to the raw_text argument. Each upper level sequence contains a sequence of tokens for each sample.

• probabilities - No probabilities should be passed for Token Classification tasks. Passing probabilities will result in an error.

Examples

Text Classification

Single Class Predictions

>>> predictions = ['class_1', 'class_1', 'class_2']
>>> probabilities = [[0.2, 0.8], [0.5, 0.5], [0.3, 0.7]]

Multilabel Predictions

>>> predictions = [[0, 0, 1], [0, 1, 1]]
>>> probabilities = [[0.2, 0.3, 0.8], [0.4, 0.9, 0.6]]

Token Classification

>>> predictions = [[('class_1', 0, 2, 0.8), ('class_2', 7, 10, 0.9)], [('class_2', 42, 54, 0.4)], []]

abstract run_logic(context) → CheckResult

Run check.

to_json(indent: int = 3, include_version: bool = True, include_defaults: bool = True) → str

Serialize check instance to JSON string.

class Suite

NLP suite to run checks of types: TrainTestCheck, SingleDatasetCheck, ModelOnlyCheck.

Methods

add(check)	Add a check or a suite to current suite.
config()	Return suite configuration (checks' conditions' configuration not yet supported).
from_config(conf[, version_unmatch])	Return suite object from a CheckConfig object.
from_json(conf[, version_unmatch])	Deserialize suite instance from JSON string.
remove(index)	Remove a check by given index.
run([train_dataset, test_dataset, ...])	Run all checks.
supported_checks()	Return tuple of supported check types of this suite.
to_json([indent])	Serialize suite instance to JSON string.

__init__(name: str, *checks: Union[BaseCheck, BaseSuite])

add(check: Union[BaseCheck, BaseSuite])

Add a check or a suite to current suite.

Parameters

checkBaseCheck

A check or suite to add.

config() → SuiteConfig

Return suite configuration (checks’ conditions’ configuration not yet supported).

Returns

SuiteConfig

includes the suite name, and list of check configs.

classmethod from_config(conf: SuiteConfig, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self

Return suite object from a CheckConfig object.

Parameters

confSuiteConfig

the SuiteConfig object

Returns

BaseSuite

the suite class object from given config

from_json(conf: str, version_unmatch: Optional[Union[Literal['raise'], Literal['warn']]] = 'warn') → Self

Deserialize suite instance from JSON string.

remove(index: int)

Remove a check by given index.

Parameters

indexint

Index of check to remove.

run(train_dataset: Optional[TextData] = None, test_dataset: Optional[TextData] = None, with_display: bool = True, train_predictions: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, test_predictions: Optional[Union[Sequence[Union[str, int]], Sequence[Sequence[Union[str, int]]], Sequence[Sequence[Tuple[str, int, int, float]]]]] = None, train_probabilities: Optional[Sequence[Sequence[float]]] = None, test_probabilities: Optional[Sequence[Sequence[float]]] = None, random_state: int = 42) → SuiteResult

Run all checks.

Parameters

train_dataset: Union[TextData, None] , default: None

TextData object, representing data an estimator was fitted on

test_dataset: Union[TextData, None] , default: None

TextData object, representing data an estimator predicts on

with_displaybool , default: True

flag that determines if checks will calculate display (redundant in some checks).

train_predictions: Union[TTextPred, None] , default: None

predictions on train dataset

test_predictions: Union[TTextPred, None] , default: None

predictions on test dataset

train_probabilities: Union[TTextProba, None] , default: None

probabilities on train dataset

test_probabilities: Union[TTextProba, None] , default: None

probabilities on test_dataset dataset

random_stateint, default 42

A seed to set for pseudo-random functions, primarily sampling.

The accepted formats for providing model predictions and probabilities are detailed below

Text Classification

Single Class Predictions

• predictions - A sequence of class names or indices with one entry per sample, matching the set of classes present in the labels.

• probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should sum to 1 for each sample.

Multilabel Predictions

• predictions - A sequence of sequences with each element containing a binary vector denoting the presence of the i-th class for the given sample. Each such vector should have one binary indicator per class according to the class (sorted) order. More than one class can be present for each sample.

• probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should range from 0 to 1 for each sample, but are not required to sum to 1.

Token Classification

• predictions - A sequence of sequences, with the inner sequence containing tuples in the following format: (class_name, span_start, span_end, class_probability). span_start and span_end are the start and end character indices of the token within the text, as it was passed to the raw_text argument. Each upper level sequence contains a sequence of tokens for each sample.

• probabilities - No probabilities should be passed for Token Classification tasks. Passing probabilities will result in an error.

Returns

SuiteResult

All results by all initialized checks

Examples

Text Classification

Single Class Predictions

>>> predictions = ['class_1', 'class_1', 'class_2']
>>> probabilities = [[0.2, 0.8], [0.5, 0.5], [0.3, 0.7]]

Multilabel Predictions

>>> predictions = [[0, 0, 1], [0, 1, 1]]
>>> probabilities = [[0.2, 0.3, 0.8], [0.4, 0.9, 0.6]]

Token Classification

>>> predictions = [[('class_1', 0, 2, 0.8), ('class_2', 7, 10, 0.9)], [('class_2', 42, 54, 0.4)], []]

classmethod supported_checks() → Tuple

Return tuple of supported check types of this suite.

to_json(indent: int = 3) → str

Serialize suite instance to JSON string.

previous

Suite.to_json

next

checks