deepchecks.nlp#
Package for nlp functionality.
Modules
Module importing all nlp checks. |
|
Module contains all prebuilt nlp suites. |
|
Module for working with pre-built datasets. |
|
Module containing metrics utils for nlp tasks. |
|
Utils package for nlp functionality. |
|
The dataset module containing the tabular Dataset class and its functions. |
|
Module for nlp base checks. |
|
Module for base nlp context. |
|
Module for base nlp suite. |
Classes
- class TextData[source]#
TextData wraps together the raw text data and the labels for the nlp task.
The TextData class contains metadata and methods intended for easily accessing metadata relevant for the training or validating of ML models.
- Parameters
- raw_textt.Sequence[str], default: None
The raw text data, a sequence of strings representing the raw text of each sample. If not given, tokenized_text must be given, and raw_text will be created from it by joining the tokens with spaces.
- tokenized_textt.Sequence[t.Sequence[str]], default: None
The tokenized text data, a sequence of sequences of strings representing the tokenized text of each sample. Only relevant for task_type ‘token_classification’. If not given, raw_text must be given, and tokenized_text will be created from it by splitting the text by spaces.
- labelt.Optional[TTextLabel], default: None
The label for the text data. Can be either a text_classification label or a token_classification label. If None, the label is not set.
text_classification label - For text classification the accepted label format differs between multilabel and single label cases. For single label data, the label should be passed as a sequence of labels, with one entry per sample that can be either a string or an integer. For multilabel data, the label should be passed as a sequence of sequences, with the sequence for each sample being a binary vector, representing the presence of the i-th label in that sample.
token_classification label - For token classification the accepted label format is the IOB format or similar to it. The Label must be a sequence of sequences of strings or integers, with each sequence corresponding to a sample in the tokenized text, and exactly the length of the corresponding tokenized text.
- task_typestr, default: None
The task type for the text data. Can be either ‘text_classification’ or ‘token_classification’. Must be set if label is provided.
- namet.Optional[str] , default: None
The name of the dataset. If None, the dataset name will be defined when running it within a check.
- metadatat.Optional[t.Union[pd.DataFrame, str]] , default: None
Metadata for the samples. Metadata must be given as a pandas DataFrame or a path to a pandas DataFrame compatible csv file, with the rows representing each sample and columns representing the different metadata columns. If None, no metadata is set. The number of rows in the metadata DataFrame must be equal to the number of samples in the dataset, and the order of the rows must be the same as the order of the samples in the dataset. For more on metadata, see the NLP Metadata Guide.
- categorical_metadatat.Optional[t.List[str]] , default: None
The names of the categorical metadata columns. If None, categorical metadata columns are automatically inferred. Only relevant if metadata is not None.
- propertiest.Optional[t.Union[pd.DataFrame, str]] , default: None
The text properties for the samples. Properties must be given as either a pandas DataFrame or a path to a pandas DataFrame compatible csv file, with the rows representing each sample and columns representing the different properties. If None, no properties are set. The number of rows in the properties DataFrame must be equal to the number of samples in the dataset, and the order of the rows must be the same as the order of the samples in the dataset. In order to calculate the default properties, use the TextData.calculate_builtin_properties function after the creation of the TextData object. For more on properties, see the NLP Properties Guide.
- categorical_propertiest.Optional[t.List[str]] , default: None
The names of the categorical properties columns. Should be given only for custom properties, not for any of the built-in properties. If None, categorical properties columns are automatically inferred for custom properties.
- embeddingst.Optional[Union[np.ndarray, pd.DataFrame, str]], default: None
The text embeddings for the samples. Embeddings must be given as a numpy array (or a path to an .npy file containing a numpy array) of shape (N, E), where N is the number of samples in the TextData object and E is the number of embeddings dimensions. The numpy array must be in the same order as the samples in the TextData. If None, no embeddings are set.
In order to use the built-in embeddings, use the TextData.calculate_builtin_embeddings function after the creation of the TextData object. For more on embeddings, see the Text Embeddings Guide
- Attributes
categorical_metadata
Return categorical metadata column names.
categorical_properties
Return categorical properties names.
embeddings
Return the embeddings of for the dataset.
label
Return the label defined in the dataset.
metadata
Return the metadata of for the dataset.
n_samples
Return number of samples in the dataset.
- name
numerical_metadata
Return numeric metadata column names.
numerical_properties
Return numerical properties names.
properties
Return the properties of the dataset.
task_type
Return the task type.
text
Return sequence of raw text samples.
tokenized_text
Return sequence of tokenized text samples.
Methods
calculate_builtin_embeddings
([model, ...])Calculate the built-in embeddings of the dataset.
calculate_builtin_properties
([...])Calculate the default properties of the dataset.
cast_to_dataset
(obj)Verify Dataset or transform to Dataset.
copy
([rows_to_use])Create a copy of this Dataset with new data.
describe
([n_properties_to_show, ...])Provide holistic view of the data.
Return the original indexes of the text samples.
get_sample_at_original_index
(index)Return the text sample at the original index.
Return True if label was set.
head
([n_samples, model_classes])Return a copy of the dataset as a pandas Dataframe with the first n_samples samples.
Check if the dataset is multi-label.
is_sampled
(n_samples)Return True if the dataset number of samples will decrease when sampled with n_samples samples.
label_for_display
([model_classes])Return the label defined in the dataset in a format that can be displayed.
label_for_print
([model_classes])Return the label defined in the dataset in a format that can be printed nicely.
len_when_sampled
(n_samples)Return number of samples in the sampled dataframe this dataset is sampled with n_samples samples.
sample
(n_samples[, replace, random_state, ...])Create a copy of the dataset object, with the internal data being a sample of the original data.
save_properties
(path)Save the dataset properties to csv.
set_embeddings
(embeddings[, verbose])Set the embeddings of the dataset.
set_metadata
(metadata[, categorical_metadata])Set the metadata of the dataset.
set_properties
(properties[, ...])Set the properties of the dataset.
validate_textdata_compatibility
(other_text_data)Verify that all provided datasets share same label name and task types.
- __init__(raw_text: Optional[Sequence[str]] = None, tokenized_text: Optional[Sequence[Sequence[str]]] = None, label: Optional[Union[Sequence[Union[int, str, Tuple[Union[int, str]]]], Sequence[Sequence[Union[str, int]]], Sequence[None]]] = None, task_type: Optional[str] = None, name: Optional[str] = None, embeddings: Optional[Union[DataFrame, ndarray, str]] = None, metadata: Optional[DataFrame] = None, categorical_metadata: Optional[List[str]] = None, properties: Optional[DataFrame] = None, categorical_properties: Optional[List[str]] = None)[source]#
- calculate_builtin_embeddings(model: str = 'miniLM', file_path: str = 'embeddings.npy', device: Optional[str] = None, long_sample_behaviour: str = 'average+warn', open_ai_batch_size: int = 500)[source]#
Calculate the built-in embeddings of the dataset.
- Parameters
- modelstr, default: ‘miniLM’
The model to use for calculating the embeddings. Possible values are: ‘miniLM’: using the miniLM model in the sentence-transformers library. ‘open_ai’: using the ADA model in the open_ai library. Requires an API key.
- file_pathstr, default: ‘embeddings.npy’
The path to save the embeddings to.
- devicestr, default: None
The device to use for calculating the embeddings. If None, the default device will be used.
- long_sample_behaviourstr, default ‘average+warn’
How to handle long samples. Averaging is done as described in https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb Currently, applies only to the ‘open_ai’ model, as the ‘miniLM’ model can handle long samples.
- Options are:
‘average+warn’ (default): average the embeddings of the chunks and warn if the sample is too long.
‘average’: average the embeddings of the chunks.
‘truncate’: truncate the sample to the maximum length.
‘raise’: raise an error if the sample is too long.
‘nan’: return an embedding vector of nans for each sample that is too long.
- open_ai_batch_sizeint, default 500
The amount of samples to send to open ai in each batch. Reduce if getting errors from open ai.
- calculate_builtin_properties(include_properties: Optional[List[str]] = None, ignore_properties: Optional[List[str]] = None, include_long_calculation_properties: bool = False, ignore_non_english_samples_for_english_properties: bool = True, device: Optional[str] = None)[source]#
Calculate the default properties of the dataset.
- Parameters
- include_propertiesList[str], default None
The properties to calculate. If None, all default properties will be calculated. Cannot be used together with ignore_properties parameter.
- ignore_propertiesList[str], default None
The properties to ignore. If None, no properties will be ignored. Cannot be used together with properties parameter.
- include_long_calculation_propertiesbool, default False
Whether to include properties that may take a long time to calculate. If False, these properties will be ignored.
- ignore_non_english_samples_for_english_propertiesbool, default True
Whether to ignore samples that are not in English when calculating English properties. If False, samples that are not in English will be calculated as well. This parameter is ignored when calculating non-English properties. English-Only properties WILL NOT work properly on non-English samples, and this parameter should be used only when you are sure that all the samples are in English.
- deviceint, default None
The device to use for the calculation. If None, the default device will be used.
- classmethod cast_to_dataset(obj: Any) TextData [source]#
Verify Dataset or transform to Dataset.
Function verifies that provided value is a non-empty instance of Dataset, otherwise raises an exception, but if the ‘cast’ flag is set to True it will also try to transform provided value to the Dataset instance.
- Parameters
- obj
value to verify
- Raises
- DeepchecksValueError
if the provided value is not a TextData instance; if the provided value cannot be transformed into Dataset instance;
- copy(rows_to_use: Optional[Sequence[int]] = None) TDataset [source]#
Create a copy of this Dataset with new data.
- Parameters
- rows_to_uset.Optional[t.List[int]] , default: None
The rows to use in the new copy. If None, the new copy will contain all the rows.
- describe(n_properties_to_show: Optional[int] = 4, properties_to_show: Optional[List[str]] = None, max_num_labels_to_show: Optional[int] = 5, model_classes: Optional[List[str]] = None)[source]#
Provide holistic view of the data.
Generates the following plots: 1. Label distribution 2. Statistics about the data such as number of samples, annotation ratio, list of metadata columns, list of text properties and so on. 3. Property distribution for the text properties defined either by n_properties_to_show or properties_to_show parameter.
- Parameters
- n_properties_to_showint, default: 4
Number of properties to consider for generating property distribution graphs. If properties_to_show is provided, this value is ignored.
- properties_to_showList[str], default: None
List of property names to consider for generating property distribution graphs. If None, all the properties are considered.
- max_num_labels_to_showint, default: 5
The threshold to display the maximum number of labels on the label distribution pie chart and display rest of the labels under “Others” category.
- model_classesOptional[List[str]], default: None
List of classes names to use for multi-label display. Only used if the dataset is multi-label.
- Returns
- Displays the Plotly Figure.
- get_original_text_indexes() Sequence[int] [source]#
Return the original indexes of the text samples.
- Returns
- t.Sequence[int]
Original indexes of the text samples.
- get_sample_at_original_index(index: int) str [source]#
Return the text sample at the original index.
- Parameters
- indexint
Original index of the text sample.
- Returns
- str
Text sample at the original index.
- head(n_samples: int = 5, model_classes: Optional[list] = None) DataFrame [source]#
Return a copy of the dataset as a pandas Dataframe with the first n_samples samples.
- Parameters
- n_samplesint, default 5
Number of samples to return.
- model_classeslist, default None
List of classes names to use for multi-label display. Only used if the dataset is multi-label.
- Returns
- pd.DataFrame
A copy of the dataset as a pandas Dataframe with the first n_samples samples.
- is_sampled(n_samples: Optional[int])[source]#
Return True if the dataset number of samples will decrease when sampled with n_samples samples.
- property label: Union[Sequence[Union[int, str, Tuple[Union[int, str]]]], Sequence[Sequence[Union[str, int]]], Sequence[None]]#
Return the label defined in the dataset.
- Returns
- TTextLabel
- label_for_display(model_classes: Optional[list] = None) Union[Sequence[Union[int, str, Tuple[Union[int, str]]]], Sequence[Sequence[Union[str, int]]], Sequence[None]] [source]#
Return the label defined in the dataset in a format that can be displayed.
- Parameters
- model_classeslist, default None
List of classes names to use for multi-label display. Only used if the dataset is multi-label.
- Returns
- TTextLabel
- label_for_print(model_classes: Optional[list] = None) List[str] [source]#
Return the label defined in the dataset in a format that can be printed nicely.
- Parameters
- model_classeslist, default None
List of classes names to use for multi-label display. Only used if the dataset is multi-label.
- Returns
- List[str]
- len_when_sampled(n_samples: Optional[int])[source]#
Return number of samples in the sampled dataframe this dataset is sampled with n_samples samples.
- sample(n_samples: int, replace: bool = False, random_state: Optional[int] = None, drop_na_label: bool = False) TDataset [source]#
Create a copy of the dataset object, with the internal data being a sample of the original data.
- Parameters
- n_samplesint
Number of samples to draw.
- replacebool, default: False
Whether to sample with replacement.
- random_statet.Optional[int] , default None
Random state.
- drop_na_labelbool, default: False
Whether to take sample only from rows with exiting label.
- Returns
- Dataset
instance of the Dataset with sampled internal dataframe.
- save_properties(path: str)[source]#
Save the dataset properties to csv.
- Parameters
- pathstr
Path to save the properties to.
- set_embeddings(embeddings: ndarray, verbose: bool = True)[source]#
Set the embeddings of the dataset.
- Parameters
- embeddingspd.DataFrame
Embeddings to set.
- verbosebool, default: True
Whether to print information about the process.
- set_metadata(metadata: DataFrame, categorical_metadata: Optional[Sequence[str]] = None)[source]#
Set the metadata of the dataset.
- set_properties(properties: DataFrame, categorical_properties: Optional[Sequence[str]] = None)[source]#
Set the properties of the dataset.
- property task_type: Optional[deepchecks.nlp.task_type.TaskType]#
Return the task type.
- Returns
- t.Optional[TaskType]
Task type
- property text: Sequence[str]#
Return sequence of raw text samples.
- Returns
- t.Sequence[str]
Sequence of raw text samples.
- property tokenized_text: Sequence[Sequence[str]]#
Return sequence of tokenized text samples.
- Returns
- t.Sequence[t.Sequence[str]]
Sequence of tokenized text samples.
- validate_textdata_compatibility(other_text_data: TextData) bool [source]#
Verify that all provided datasets share same label name and task types.
- Parameters
- other_text_dataTextData
The other dataset TextData object to compare with.
- Returns
- bool
True if provided dataset share same label name and task types.
- class Context[source]#
Contains all the data + properties the user has passed to a check/suite, and validates it seamlessly.
- Parameters
- train_datasetUnion[TextData, None] , default: None
TextData object, representing data an estimator was fitted on
- test_datasetUnion[TextData, None] , default: None
TextData object, representing data an estimator predicts on
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- train_predUnion[TTextPred, None] , default: None
predictions on train dataset
- test_predUnion[TTextPred, None] , default: None
predictions on test dataset
- train_probaUnion[TTextProba, None] , default: None
probabilities on train dataset
- test_probaUnion[TTextProba, None] , default: None
probabilities on test dataset
- model_classesOptional[List] , default: None
list of classes known to the model
- random_state: int, default 42
A seed to set for pseudo-random functions , primarily sampling.
- Attributes
common_datasets_properties
Return common for train and test datasets properties.
model
Return model if exists, otherwise raise error.
model_classes
Return ordered list of possible label classes for classification tasks.
model_name
Return the name of the model.
observed_classes
Return the observed classes in both train and test.
task_type
Return task type if model & train_dataset & label exists.
test
Return test if exists, otherwise raise error.
train
Return train if exists, otherwise raise error.
with_display
Return the with_display flag.
Methods
assert_task_type
(*expected_types)Assert task_type matching given types.
finalize_check_result
(check_result, check[, ...])Run final processing on a check result which includes validation, conditions processing and sampling footnote.
get_data_by_kind
(kind)Return the relevant Dataset by given kind.
get_scorers
([scorers, use_avg_defaults])Return initialized & validated scorers if provided or default scorers otherwise.
get_single_scorer
([scorer, use_avg_defaults])Return initialized & validated scorer if provided or a default scorer otherwise.
Return whether there is test_dataset dataset defined.
infer_task_type
(train_dataset, test_dataset)Infer the task type.
Return whether the task is multi-label classification.
raise_if_multi_label_task
([check])Raise an exception if it is a multi-label classification task.
raise_if_token_classification_task
([check])Raise an exception if it is a token classification task.
- __init__(train_dataset: Optional[TextData] = None, test_dataset: Optional[TextData] = None, with_display: bool = True, train_pred: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, test_pred: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, train_proba: Optional[Sequence[Sequence[float]]] = None, test_proba: Optional[Sequence[Sequence[float]]] = None, model_classes: Optional[List] = None, random_state: int = 42)[source]#
- assert_task_type(*expected_types: TaskType)[source]#
Assert task_type matching given types.
If task_type is defined, validate it and raise error if needed, else returns True. If task_type is not defined, return False.
- property common_datasets_properties: Dict[str, str]#
Return common for train and test datasets properties.
- finalize_check_result(check_result, check, dataset_kind: Optional[DatasetKind] = None)[source]#
Run final processing on a check result which includes validation, conditions processing and sampling footnote.
- get_data_by_kind(kind: DatasetKind)[source]#
Return the relevant Dataset by given kind.
- get_scorers(scorers: Optional[Union[Mapping[str, Union[str, Callable]], List[str]]] = None, use_avg_defaults=True) List[DeepcheckScorer] [source]#
Return initialized & validated scorers if provided or default scorers otherwise.
- Parameters
- scorersUnion[List[str], Dict[str, Union[str, Callable]]], default: None
List of scorers to use. If None, use default scorers. Scorers can be supplied as a list of scorer names or as a dictionary of names and functions.
- use_avg_defaultsbool, default: True
If no scorers were provided, for classification, determines whether to use default scorers that return an averaged metric, or default scorers that return a metric per class.
- Returns
- ——-
- List[DeepcheckScorer]
A list of initialized & validated scorers.
- get_single_scorer(scorer: Optional[Union[Mapping[str, Union[str, Callable]], List[str]]] = None, use_avg_defaults=True) DeepcheckScorer [source]#
Return initialized & validated scorer if provided or a default scorer otherwise.
- Parameters
- scorert.Union[t.Mapping[str, t.Union[str, t.Callable]], t.List[str], None], default: None
List of scorers to use. If None, use default scorers. Scorers can be supplied as a list of scorer names or as a dictionary of names and functions.
- use_avg_defaultsbool, default: True
If no scorers were provided, for classification, determines whether to use default scorers that return an averaged metric, or default scorers that return a metric per class.
- Returns
- ——-
- List[DeepcheckScorer]
An initialized & validated scorer.
- static infer_task_type(train_dataset: TextData, test_dataset: TextData)[source]#
Infer the task type.
- property model: deepchecks.nlp.context._DummyModel#
Return model if exists, otherwise raise error.
- property model_classes: List#
Return ordered list of possible label classes for classification tasks.
- property observed_classes: List#
Return the observed classes in both train and test.
- raise_if_multi_label_task(check=None)[source]#
Raise an exception if it is a multi-label classification task.
- raise_if_token_classification_task(check=None)[source]#
Raise an exception if it is a token classification task.
- property task_type: deepchecks.nlp.task_type.TaskType#
Return task type if model & train_dataset & label exists. otherwise, raise error.
- property test#
Return test if exists, otherwise raise error.
- property train#
Return train if exists, otherwise raise error.
- class SingleDatasetCheck[source]#
Parent class for checks that only use one dataset.
Methods
add_condition
(name, condition_func, **params)Add new condition function to the check.
Remove all conditions from this check instance.
conditions_decision
(result)Run conditions on given result.
config
([include_version, include_defaults])Return check configuration (conditions' configuration not yet supported).
alias of
Context
from_config
(conf[, version_unmatch])Return check object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize check instance from JSON string.
metadata
([with_doc_link])Return check metadata.
name
()Name of class in split camel case.
params
([show_defaults])Return parameters to show when printing the check.
remove_condition
(index)Remove given condition by index.
run
(dataset[, model, with_display, ...])Run check.
run_logic
(context, dataset_kind)Run check.
to_json
([indent, include_version, ...])Serialize check instance to JSON string.
- add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#
Add new condition function to the check.
- Parameters
- namestr
Name of the condition. should explain the condition action and parameters
- condition_funcCallable[[Any], Union[List[ConditionResult], bool]]
Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
- paramsdict
Additional parameters to pass when calling the condition function.
- conditions_decision(result: CheckResult) List[ConditionResult] [source]#
Run conditions on given result.
- config(include_version: bool = True, include_defaults: bool = True) CheckConfig [source]#
Return check configuration (conditions’ configuration not yet supported).
- Returns
- CheckConfig
includes the checks class name, params, and module name.
- classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return check object from a CheckConfig object.
- Parameters
- confDict[Any, Any]
- Returns
- BaseCheck
the check class object from given config
- classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize check instance from JSON string.
- metadata(with_doc_link: bool = False) CheckMetadata [source]#
Return check metadata.
- Parameters
- with_doc_linkbool, default False
whethere to include doc link in summary or not
- Returns
- Dict[str, Any]
- params(show_defaults: bool = False) Dict [source]#
Return parameters to show when printing the check.
- remove_condition(index: int)[source]#
Remove given condition by index.
- Parameters
- indexint
index of condtion to remove
- run(dataset: TextData, model=None, with_display: bool = True, predictions: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, probabilities: Optional[Sequence[Sequence[float]]] = None, model_classes: Optional[List] = None, random_state: int = 42) CheckResult [source]#
Run check.
- Parameters
- dataset: TextData
Dataset representing data an estimator was fitted on
- model: object, default: None
Dummy model object for compatibility with SingleDatasetBaseCheck
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- predictions: Union[TTextPred, None] , default: None
predictions on dataset
- probabilities: Union[TTextProba, None] , default: None
probabilities on dataset
- model_classes: Optional[List], default: None
For classification: list of classes known to the model
- random_stateint, default 42
A seed to set for pseudo-random functions, primarily sampling.
Notes
The accepted formats for providing model predictions and probabilities are detailed below
Text Classification
Single Class Predictions
predictions - A sequence of class names or indices with one entry per sample, matching the set of classes present in the labels.
probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should sum to 1 for each sample.
Multilabel Predictions
predictions - A sequence of sequences with each element containing a binary vector denoting the presence of the i-th class for the given sample. Each such vector should have one binary indicator per class according to the class (sorted) order. More than one class can be present for each sample.
probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should range from 0 to 1 for each sample, but are not required to sum to 1.
Token Classification
predictions - A sequence of sequences, with the inner sequence containing tuples in the following format: (class_name, span_start, span_end, class_probability). span_start and span_end are the start and end character indices of the token within the text, as it was passed to the raw_text argument. Each upper level sequence contains a sequence of tokens for each sample.
probabilities - No probabilities should be passed for Token Classification tasks. Passing probabilities will result in an error.
Examples
Text Classification
Single Class Predictions
>>> predictions = ['class_1', 'class_1', 'class_2'] >>> probabilities = [[0.2, 0.8], [0.5, 0.5], [0.3, 0.7]]
Multilabel Predictions
>>> predictions = [[0, 0, 1], [0, 1, 1]] >>> probabilities = [[0.2, 0.3, 0.8], [0.4, 0.9, 0.6]]
Token Classification
>>> predictions = [[('class_1', 0, 2, 0.8), ('class_2', 7, 10, 0.9)], [('class_2', 42, 54, 0.4)], []]
- abstract run_logic(context, dataset_kind) CheckResult [source]#
Run check.
- class TrainTestCheck[source]#
Parent class for checks that compare two datasets.
The class checks train dataset and test_dataset dataset for model training and test_dataset.
Methods
add_condition
(name, condition_func, **params)Add new condition function to the check.
Remove all conditions from this check instance.
conditions_decision
(result)Run conditions on given result.
config
([include_version, include_defaults])Return check configuration (conditions' configuration not yet supported).
alias of
Context
from_config
(conf[, version_unmatch])Return check object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize check instance from JSON string.
metadata
([with_doc_link])Return check metadata.
name
()Name of class in split camel case.
params
([show_defaults])Return parameters to show when printing the check.
remove_condition
(index)Remove given condition by index.
run
(train_dataset, test_dataset[, model, ...])Run check.
run_logic
(context)Run check.
to_json
([indent, include_version, ...])Serialize check instance to JSON string.
- add_condition(name: str, condition_func: Callable[[Any], Union[ConditionResult, bool]], **params)[source]#
Add new condition function to the check.
- Parameters
- namestr
Name of the condition. should explain the condition action and parameters
- condition_funcCallable[[Any], Union[List[ConditionResult], bool]]
Function which gets the value of the check and returns object of List[ConditionResult] or boolean.
- paramsdict
Additional parameters to pass when calling the condition function.
- conditions_decision(result: CheckResult) List[ConditionResult] [source]#
Run conditions on given result.
- config(include_version: bool = True, include_defaults: bool = True) CheckConfig [source]#
Return check configuration (conditions’ configuration not yet supported).
- Returns
- CheckConfig
includes the checks class name, params, and module name.
- classmethod from_config(conf: ~deepchecks.core.checks.CheckConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return check object from a CheckConfig object.
- Parameters
- confDict[Any, Any]
- Returns
- BaseCheck
the check class object from given config
- classmethod from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize check instance from JSON string.
- metadata(with_doc_link: bool = False) CheckMetadata [source]#
Return check metadata.
- Parameters
- with_doc_linkbool, default False
whethere to include doc link in summary or not
- Returns
- Dict[str, Any]
- params(show_defaults: bool = False) Dict [source]#
Return parameters to show when printing the check.
- remove_condition(index: int)[source]#
Remove given condition by index.
- Parameters
- indexint
index of condtion to remove
- run(train_dataset: TextData, test_dataset: TextData, model=None, with_display: bool = True, train_predictions: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, test_predictions: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, train_probabilities: Optional[Sequence[Sequence[float]]] = None, test_probabilities: Optional[Sequence[Sequence[float]]] = None, model_classes: Optional[List] = None, random_state: int = 42) CheckResult [source]#
Run check.
- Parameters
- train_dataset: Union[TextData, None] , default: None
TextData object, representing data an estimator was fitted on
- test_dataset: Union[TextData, None] , default: None
TextData object, representing data an estimator predicts on
- model: object, default: None
Dummy model object for compatibility with TrainTestBaseCheck
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- train_predictions: Union[TTextPred, None] , default: None
predictions on train dataset
- test_predictions: Union[TTextPred, None] , default: None
predictions on test_dataset dataset
- train_probabilities: Union[TTextProba, None] , default: None
probabilities on train dataset
- test_probabilities: Union[TTextProba, None] , default: None
probabilities on test_dataset dataset
- model_classes: Optional[List], default: None
For classification: list of classes known to the model
- random_stateint, default 42
A seed to set for pseudo-random functions, primarily sampling.
Notes
The accepted formats for providing model predictions and probabilities are detailed below
Text Classification
Single Class Predictions
predictions - A sequence of class names or indices with one entry per sample, matching the set of classes present in the labels.
probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should sum to 1 for each sample.
Multilabel Predictions
predictions - A sequence of sequences with each element containing a binary vector denoting the presence of the i-th class for the given sample. Each such vector should have one binary indicator per class according to the class (sorted) order. More than one class can be present for each sample.
probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should range from 0 to 1 for each sample, but are not required to sum to 1.
Token Classification
predictions - A sequence of sequences, with the inner sequence containing tuples in the following format: (class_name, span_start, span_end, class_probability). span_start and span_end are the start and end character indices of the token within the text, as it was passed to the raw_text argument. Each upper level sequence contains a sequence of tokens for each sample.
probabilities - No probabilities should be passed for Token Classification tasks. Passing probabilities will result in an error.
Examples
Text Classification
Single Class Predictions
>>> predictions = ['class_1', 'class_1', 'class_2'] >>> probabilities = [[0.2, 0.8], [0.5, 0.5], [0.3, 0.7]]
Multilabel Predictions
>>> predictions = [[0, 0, 1], [0, 1, 1]] >>> probabilities = [[0.2, 0.3, 0.8], [0.4, 0.9, 0.6]]
Token Classification
>>> predictions = [[('class_1', 0, 2, 0.8), ('class_2', 7, 10, 0.9)], [('class_2', 42, 54, 0.4)], []]
- abstract run_logic(context) CheckResult [source]#
Run check.
- class Suite[source]#
NLP suite to run checks of types: TrainTestCheck, SingleDatasetCheck, ModelOnlyCheck.
Methods
add
(check)Add a check or a suite to current suite.
config
()Return suite configuration (checks' conditions' configuration not yet supported).
from_config
(conf[, version_unmatch])Return suite object from a CheckConfig object.
from_json
(conf[, version_unmatch])Deserialize suite instance from JSON string.
remove
(index)Remove a check by given index.
run
([train_dataset, test_dataset, ...])Run all checks.
Return tuple of supported check types of this suite.
to_json
([indent])Serialize suite instance to JSON string.
- add(check: Union[BaseCheck, BaseSuite])[source]#
Add a check or a suite to current suite.
- Parameters
- checkBaseCheck
A check or suite to add.
- config() SuiteConfig [source]#
Return suite configuration (checks’ conditions’ configuration not yet supported).
- Returns
- SuiteConfig
includes the suite name, and list of check configs.
- classmethod from_config(conf: ~deepchecks.core.suite.SuiteConfig, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Return suite object from a CheckConfig object.
- Parameters
- confSuiteConfig
the SuiteConfig object
- Returns
- BaseSuite
the suite class object from given config
- from_json(conf: str, version_unmatch: ~typing.Optional[~typing.Union[typing_extensions.Literal[raise], typing_extensions.Literal[warn]]] = 'warn') Self [source]#
Deserialize suite instance from JSON string.
- remove(index: int)[source]#
Remove a check by given index.
- Parameters
- indexint
Index of check to remove.
- run(train_dataset: Optional[TextData] = None, test_dataset: Optional[TextData] = None, with_display: bool = True, train_predictions: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, test_predictions: Optional[Union[Sequence[int], Sequence[str], Sequence[Sequence[int]], Sequence[Sequence[str]]]] = None, train_probabilities: Optional[Sequence[Sequence[float]]] = None, test_probabilities: Optional[Sequence[Sequence[float]]] = None, model_classes: Optional[List] = None, random_state: int = 42) SuiteResult [source]#
Run all checks.
- Parameters
- train_dataset: Union[TextData, None] , default: None
TextData object, representing data an estimator was fitted on
- test_dataset: Union[TextData, None] , default: None
TextData object, representing data an estimator predicts on
- with_displaybool , default: True
flag that determines if checks will calculate display (redundant in some checks).
- train_predictions: Union[TTextPred, None] , default: None
predictions on train dataset
- test_predictions: Union[TTextPred, None] , default: None
predictions on test dataset
- train_probabilities: Union[TTextProba, None] , default: None
probabilities on train dataset
- test_probabilities: Union[TTextProba, None] , default: None
probabilities on test_dataset dataset
- model_classes: Optional[List], default: None
For classification: list of classes known to the model
- random_stateint, default 42
A seed to set for pseudo-random functions, primarily sampling.
- Returns
- SuiteResult
All results by all initialized checks
Notes
The accepted formats for providing model predictions and probabilities are detailed below
Text Classification
Single Class Predictions
predictions - A sequence of class names or indices with one entry per sample, matching the set of classes present in the labels.
probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should sum to 1 for each sample.
Multilabel Predictions
predictions - A sequence of sequences with each element containing a binary vector denoting the presence of the i-th class for the given sample. Each such vector should have one binary indicator per class according to the class (sorted) order. More than one class can be present for each sample.
probabilities - A sequence of sequences with each element containing the vector of class probabilities for each sample. Each such vector should have one probability per class according to the class (sorted) order, and the probabilities should range from 0 to 1 for each sample, but are not required to sum to 1.
Token Classification
predictions - A sequence of sequences, with the inner sequence containing tuples in the following format: (class_name, span_start, span_end, class_probability). span_start and span_end are the start and end character indices of the token within the text, as it was passed to the raw_text argument. Each upper level sequence contains a sequence of tokens for each sample.
probabilities - No probabilities should be passed for Token Classification tasks. Passing probabilities will result in an error.
Examples
Text Classification
Single Class Predictions
>>> predictions = ['class_1', 'class_1', 'class_2'] >>> probabilities = [[0.2, 0.8], [0.5, 0.5], [0.3, 0.7]]
Multilabel Predictions
>>> predictions = [[0, 0, 1], [0, 1, 1]] >>> probabilities = [[0.2, 0.3, 0.8], [0.4, 0.9, 0.6]]
Token Classification
>>> predictions = [[('class_1', 0, 2, 0.8), ('class_2', 7, 10, 0.9)], [('class_2', 42, 54, 0.4)], []]