.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "nlp/auto_tutorials/quickstarts/plot_multi_label_classification.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_nlp_auto_tutorials_quickstarts_plot_multi_label_classification.py: .. _nlp__multilabel_quickstart: Multi Label Classification Quickstart ************************************* Deepchecks NLP tests your models during model development/research and before deploying to production. Using our testing package reduces model failures and saves tests development time. In this quickstart guide, you will learn how to use the deepchecks NLP package to analyze and evaluate a text multi label classification task. If you are interested in a regular multiclass classification task, you can refer to our :ref:`Multiclass Quickstart `. We will cover the following: 1. `Creating a TextData object and auto calculating properties <#setting-up>`__ 2. `Running the built-in suites <#running-the-deepchecks-default-suites>`__ 3. `Running individual checks <#running-individual-checks>`__ To run deepchecks for NLP, you need the following for both your train and test data: 1. Your text data - a list of strings, each string is a single sample (can be a sentence, paragraph, document, etc.). 2. Your labels and prediction in the :ref:`correct format ` (Optional). 3. :ref:`Metadata `, :ref:`Properties ` or :ref:`Embeddings ` for the provided text data (Optional). If you don't have deepchecks installed yet: .. code:: python import sys !{sys.executable} -m pip install 'deepchecks[nlp]' -U --quiet #--user Some properties calculated by ``deepchecks.nlp`` require additional packages to be installed. You can install them by running: .. code:: python import sys !{sys.executable} -m pip install 'deepchecks[nlp-properties]' -U --quiet #--user Setting Up ========== Load Data --------- For the purpose of this guide, we'll use a small subset of the `just dance `__ comment analysis dataset. A dataset containing comments, metadata and labels for a multilabel category classification use case on youtube comments. .. GENERATED FROM PYTHON SOURCE LINES 50-59 .. code-block:: default from deepchecks.nlp import TextData from deepchecks.nlp.datasets.classification import just_dance_comment_analysis data = just_dance_comment_analysis.load_data(data_format='DataFrame', as_train_test=False) metadata_cols = ['likes', 'dateComment'] data.head(2) .. rst-class:: sphx-glr-script-out .. code-block:: none include_properties and include_embeddings are incompatible with data_format="Dataframe". loading only original text data. .. raw:: html
Aesthetics and Appeal Affect and Emotion Anticipation Bodily image and Appearance Comfort ... Usability User Differences dateComment likes originalText
0 0.0 1.0 0.0 0.0 0.0 ... 0.0 0.0 2009-07-12 1.0 Thanks, Leeanna made it. and put it on my chan...
1 0.0 1.0 0.0 0.0 0.0 ... 1.0 0.0 2009-07-12 1.0 omg i love this video ! \nthanks for uploading...

2 rows × 45 columns



.. GENERATED FROM PYTHON SOURCE LINES 60-66 Create TextData Objects ------------------------ Deepchecks' :ref:`TextData ` object contains the text samples, labels, and possibly also properties and metadata. It stores cache to save time between repeated computations and contains functionalities for input validations and sampling. .. GENERATED FROM PYTHON SOURCE LINES 66-73 .. code-block:: default label_cols = data.drop(columns=['originalText'] + metadata_cols) class_names = label_cols.columns.to_list() dataset = TextData(data['originalText'], label=label_cols.to_numpy().astype(int), task_type='text_classification', metadata=data[metadata_cols], categorical_metadata=[]) .. GENERATED FROM PYTHON SOURCE LINES 74-82 Calculating Properties ---------------------- Some of deepchecks' checks use properties of the text samples for various calculations. Deepcheck has a wide variety of such properties, some simple and some that rely on external models and are more heavy to run. In order for deepchecks' checks to be able to access the properties, they must be stored within the :ref:`TextData ` object. You can read more about properties in the :ref:`Property Guide `. .. GENERATED FROM PYTHON SOURCE LINES 82-93 .. code-block:: default # properties can be either calculated directly by Deepchecks # or imported from other sources in appropriate format # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # dataset.calculate_builtin_properties(include_long_calculation_properties=True, device=device) properties = just_dance_comment_analysis.load_properties(as_train_test=False) dataset.set_properties(properties, categorical_properties=['Language']) dataset.properties.head(2) .. raw:: html
Language Text Length Average Word Length Max Word Length % Special Characters ... Formality Lexical Density Unique Noun Count Readability Score Average Sentence Length
0 en 97 4.105263 9 0.041237 ... 0.049017 89.47 7.0 102.449 6.0
1 NaN 73 3.866667 10 0.082192 ... NaN 100.00 NaN NaN 3.0

2 rows × 14 columns



.. GENERATED FROM PYTHON SOURCE LINES 94-111 Running the deepchecks default suites ===================================== Deepchecks comes with a set of pre-built suites that can be used to run a set of checks on your data, alongside with their default conditions and thresholds. You can read more about customizing and creating your own suites in the :ref:`Customizations Guide `. In this guide we'll be using 3 suites - the data integrity suite, the train test validation suite and the model evaluation suite. You can also run all the checks at once using the :mod:`full_suite `. Data Integrity -------------- We will start by doing preliminary integrity check to validate the text formatting. It is recommended to do this step before your train and test/validation splits and model training as it may imply additional data engineering is required. We'll do that using the :mod:`data_integrity ` pre-built suite. Note that we are limiting the number of samples to 1000 in order to get quick high level overview of potential issues. .. GENERATED FROM PYTHON SOURCE LINES 111-117 .. code-block:: default from deepchecks.nlp.suites import data_integrity data_integrity_suite = data_integrity(n_samples=1000) data_integrity_suite.run(dataset, model_classes=class_names) .. rst-class:: sphx-glr-script-out .. code-block:: none Data Integrity Suite: | | 0/9 [Time: 00:00] Data Integrity Suite: |█ | 1/9 [Time: 00:01, Check=Text Property Outliers] Data Integrity Suite: |██ | 2/9 [Time: 00:01, Check=Unknown Tokens] Parameter n_top_properties is set to 10 to avoid long computation time. This means that the check will run on 10 properties selected at random. If you want to run on all properties, set n_top_properties to None. Alternatively, you can set parameter properties to a list of the specific properties you want to run on. Data Integrity Suite: |████ | 4/9 [Time: 00:01, Check=Under Annotated Meta Data Segments] Data Integrity Suite: |██████ | 6/9 [Time: 00:01, Check=Conflicting Labels] Data Integrity Suite: |███████ | 7/9 [Time: 00:02, Check=Text Duplicates] Data Integrity Suite: |█████████| 9/9 [Time: 00:02, Check=Frequent Substrings] .. raw:: html
Data Integrity Suite


.. GENERATED FROM PYTHON SOURCE LINES 118-138 Integrity #1: Unknown Tokens ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ First up (in the “Didn’t Pass” tab) we see that the Unknown Tokens check has returned a problem. Looking at the result, we can see that it assumed (by default) that we’re going to use the bert-base-uncased tokenizer for our NLP model, and that if that’s the case there are many words in the dataset that contain characters (specifically here emojis) that are unrecognized by the tokenizer. This is an important insight, as bert tokenizers are very common. Integrity #2: Conflicting Labels ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Looking at the Conflicting Labels check result (in the “Didn’t Pass” tab) we can see that there are 2 occurrences of duplicate samples that have different labels. This may suggest a more severe labeling error in the dataset which we would want to explore further. .. GENERATED FROM PYTHON SOURCE LINES 140-150 Train Test Validation --------------------- The next suite, the :mod:`train_test_validation ` suite serves to validate our split and compare the two dataset. These splits can be either you training and val / test sets, in which case you'd want to run this suite after the split was made but before training, or for example your training and inference data, in which case the suite is useful for validating that the inference data is similar enough to the training data. To run this suite we'll split the data into train and test/validation sets. We'll use a predefined split based on comment dates. .. GENERATED FROM PYTHON SOURCE LINES 150-159 .. code-block:: default from deepchecks.nlp.suites import train_test_validation train_ds, test_ds = just_dance_comment_analysis.load_data( data_format='TextData', as_train_test=True, include_embeddings=True, include_properties=True) train_test_validation(n_samples=1000).run(train_ds, test_ds, model_classes=class_names) .. rst-class:: sphx-glr-script-out .. code-block:: none Train Test Validation Suite: | | 0/4 [Time: 00:00] Train Test Validation Suite: |█▎ | 1/4 [Time: 00:01, Check=Property Drift] Train Test Validation Suite: |██▌ | 2/4 [Time: 00:01, Check=Label Drift] n_jobs value -1 overridden to 1 by setting random_state. Use no seed for parallelism. n_jobs value -1 overridden to 1 by setting random_state. Use no seed for parallelism. Train Test Validation Suite: |███▊ | 3/4 [Time: 00:26, Check=Text Embeddings Drift] Train Test Validation Suite: |█████| 4/4 [Time: 00:26, Check=Train Test Samples Mix] .. raw:: html
Train Test Validation Suite


.. GENERATED FROM PYTHON SOURCE LINES 160-184 Train Test Validation #1: Properties Drift ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Based on the different properties we have calculated for the dataset, we can now search for properties whose distribution changes between the train and test datasets. Changes like this are especially important to look for when monitoring your model over time, as data drift is one of the top reasons why machine learning model’s performance degrades over time. In our case, we can see that the “% Special Characters” and the "Formality" property have different distributions between train and test. Drilling further into the results, we can see that the language of the comments in the test set is much less formal and includes more special characters (possibly emojis?) than the train set. Since this change is quite significant, we may want to consider adding more informal comments containing special characters to the train set before training (or retraining) our model. Train Test Validation #2: Embedding Drift ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Similarly to the properties drift, we can also look for embedding drift between the train and test datasets. The benefit of using embedding on top of the properties is that they are able to detect semantic changes in the data. In our case, we see there are significant semantic differences between the train and test sets. Specifically, we can see some clusters that distinctly contain more samples from train or more samples from the train dataset or more sample from the test dataset. By hovering over the clusters we can read the user comments understand what is the difference between the clusters. .. GENERATED FROM PYTHON SOURCE LINES 186-191 Model Evaluation ---------------- The suite below, the :mod:`model_evaluation ` suite, is designed to be run after a model has been trained and requires model predictions which can be supplied via the relevant arguments in the ``run`` function. .. GENERATED FROM PYTHON SOURCE LINES 191-210 .. code-block:: default train_preds, test_preds = just_dance_comment_analysis.\ load_precalculated_predictions(pred_format='predictions', as_train_test=True) train_probas, test_probas = just_dance_comment_analysis.\ load_precalculated_predictions(pred_format='probabilities', as_train_test=True) from deepchecks.nlp.suites import model_evaluation suite = model_evaluation(n_samples=1000) result = suite.run(train_ds, test_ds, train_predictions=train_preds, test_predictions=test_preds, train_probabilities=train_probas, test_probabilities=test_probas, model_classes=class_names) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Model Evaluation Suite: | | 0/4 [Time: 00:00] Model Evaluation Suite: |█▎ | 1/4 [Time: 00:00, Check=Prediction Drift] Model Evaluation Suite: |██▌ | 2/4 [Time: 00:00, Check=Train Test Performance]Parameter n_top_properties is set to 10 to avoid long computation time. This means that the check will run on 10 properties selected at random. If you want to run on all properties, set n_top_properties to None. Alternatively, you can set parameter properties to a list of the specific properties you want to run on. Parameter n_top_properties is set to 10 to avoid long computation time. This means that the check will run on 10 properties selected at random. If you want to run on all properties, set n_top_properties to None. Alternatively, you can set parameter properties to a list of the specific properties you want to run on. Model Evaluation Suite: |███▊ | 3/4 [Time: 00:21, Check=Property Segments Performance] Model Evaluation Suite: |█████| 4/4 [Time: 00:21, Check=Metadata Segments Performance] .. raw:: html
Model Evaluation Suite


.. GENERATED FROM PYTHON SOURCE LINES 211-219 Model Eval #1: Train Test Performance ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We can immediately see in the "Didn't Pass" tab that there has been significant degradation in the Recall on class “Pain and Discomfort”. Moreover, it seems there is a general deterioration in our model performance on the test set compared to the train set. This can be explained based on the data drift we saw in the previous suite. .. GENERATED FROM PYTHON SOURCE LINES 221-229 Running Individual Checks ========================= Checks can also be run individually as well as within a suite. You can learn more about customizing suites, checks and conditions in our :ref:`Customizations Guide `. In this section, we'll show you how to do that while showcasing one of our most interesting checks - :ref:`PropertySegmentPerformance `. .. GENERATED FROM PYTHON SOURCE LINES 229-237 .. code-block:: default from deepchecks.nlp.checks import PropertySegmentsPerformance check = PropertySegmentsPerformance(segment_minimum_size_ratio=0.05) check = check.add_condition_segments_relative_performance_greater_than(0.1) result = check.run(test_ds, probabilities=test_probas) result.show() .. rst-class:: sphx-glr-script-out .. code-block:: none Parameter n_top_properties is set to 10 to avoid long computation time. This means that the check will run on 10 properties selected at random. If you want to run on all properties, set n_top_properties to None. Alternatively, you can set parameter properties to a list of the specific properties you want to run on. .. raw:: html
Property Segments Performance