UnknownTokens#
- class UnknownTokens[source]#
Find samples that contain tokens unsupported by your tokenizer.
- Parameters
- tokenizer: t.Any , default: None
Tokenizer from the HuggingFace transformers library to use for tokenization. If None, AutoTokenizer.from_pretrained(‘bert-base-uncased’) will be used.
- group_singleton_words: bool, default: False
If True, group all words that appear only once in the data into the “Other” category in the display.
- n_most_commonint , default: 5
Number of most common words with unknown tokens to show in the display.
- n_samples: int, default: 1_000_000
number of samples to use for this check.
- random_stateint, default: 42
random seed for all check internals.
- max_text_length_for_displayint, default 30
truncate text samples to given length before display.
- __init__(tokenizer: Optional[Any] = None, group_singleton_words: bool = False, n_most_common: int = 5, n_samples: int = 1000000, random_state: int = 42, max_text_length_for_display: int = 30, **kwargs)[source]#
- __new__(*args, **kwargs)#
Methods
|
Add new condition function to the check. |
|
Add condition that checks if the ratio of unknown words is less than a given ratio. |
Remove all conditions from this check instance. |
|
Run conditions on given result. |
|
|
Return check configuration (conditions' configuration not yet supported). |
Create pie chart with most common unknown words. |
|
|
Find words with unknown tokens in samples. |
|
Return check object from a CheckConfig object. |
|
Deserialize check instance from JSON string. |
|
Return check metadata. |
Name of class in split camel case. |
|
|
Return parameters to show when printing the check. |
Remove given condition by index. |
|
|
Run check. |
|
Run check. |
|
Serialize check instance to JSON string. |
