Weights & Biases (wandb)#

In this guide, we will demonstrate how to export the Check’s and the Suite’s output to wandb.

This enables to view the exported results in wandb and compare the results you receive with different parameters or model/data.

Structure:

Load Data#

from deepchecks.tabular.datasets.classification import iris

train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()

Run a Check#

from deepchecks.tabular.checks import WholeDatasetDrift

result = WholeDatasetDrift().add_condition_overall_drift_value_less_than(
).run(train_dataset, test_dataset, model)

Observe CheckResult Display and Value#

result
Whole Dataset Drift


the value can be observed

result.value

Out:

{'domain_classifier_auc': 0.4545454545454546, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}

Export a Check’s Output (CheckResult) to wandb#

Exporting the output to wandb is possible using the to_wandb function. This function exports the check outputs to a wandb project.

The output display that will be export will be a bit different from what you usually see. Only the tables and the plots are being exported.

On default if you export a single check to wandb without a wandb run active it will create a project with the name deepchecks and the and the check’s metadata in the config and export the results there.

See Check to_wandb options#

from deepchecks.core import CheckResult

help(CheckResult.to_wandb)

Out:

Help on function to_wandb in module deepchecks.core.check_result:

to_wandb(self, dedicated_run: Optional[bool] = None, **kwargs)
    Send result to wandb.

    Parameters
    ----------
    dedicated_run : bool, default True
        whether to create a separate wandb run or not
        (deprecated parameter, does not have any effect anymore)
    kwargs: Keyword arguments to pass to wandb.init.
            Default project name is deepchecks.
            Default config is the check metadata (params, train/test/ name etc.).

To use this function you first need to login to your wandb account

import wandb

wandb.login()

Out:

False
result.to_wandb()

Out:

wandb: Tracking run with wandb version 0.12.18
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20220619_121538-2wd6bmq3
wandb: Find logs at: ./wandb/offline-run-20220619_121538-2wd6bmq3/logs

Exporting a Suite’s Output (SuiteResult) to wandb#

Run Suite and export to wandb#

Exporting a suite to wandb is the same as exporting a check. The display in wand will have a section per check.

On default if you export a suite to wandb without a wandb run active it will create a project with the name deepchecks and the suite’s name in the config and send the results there.

from deepchecks.tabular.suites import full_suite

suite = full_suite()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.to_wandb()

Out:

Full Suite:
|                                    | 0/36 [00:00<?, ? Check/s]
Full Suite:
|#                                   | 1/36 [00:00<00:08,  3.95 Check/s, Check=Performance Report]
Full Suite:
|####                                | 4/36 [00:01<00:10,  3.13 Check/s, Check=Segment Performance]
Full Suite:
|######                              | 6/36 [00:01<00:06,  4.69 Check/s, Check=Simple Model Comparison]Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327

Full Suite:
|#######                             | 7/36 [00:01<00:06,  4.77 Check/s, Check=Model Error Analysis]
Full Suite:
|#############                       | 13/36 [00:01<00:01, 12.39 Check/s, Check=Model Inference Time]
Full Suite:
|#######################             | 23/36 [00:01<00:00, 26.20 Check/s, Check=Feature Label Correlation Change]
Full Suite:
|############################        | 28/36 [00:02<00:00, 22.48 Check/s, Check=Special Characters]
Full Suite:
|####################################| 36/36 [00:02<00:00, 29.11 Check/s, Check=Feature Label Correlation]

wandb: Tracking run with wandb version 0.12.18
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20220619_121546-2lfciik3
wandb: Find logs at: ./wandb/offline-run-20220619_121546-2lfciik3/logs

You can also set all the kwargs the wandb.init will get:

suite_result.to_wandb(project='my-suite-project', config={'suite-name': 'full-suite'})

Out:

wandb: Tracking run with wandb version 0.12.18
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20220619_121551-c1ms79v0
wandb: Find logs at: ./wandb/offline-run-20220619_121551-c1ms79v0/logs

Total running time of the script: ( 0 minutes 21.267 seconds)

Gallery generated by Sphinx-Gallery