Weights & Biases (wandb)

In this guide, we will demonstrate how to export the Check’s and the Suite’s output to wandb.

This enables to view the exported results in wandb and compare the results you receive with different parameters or model/data.

Structure:

• Load Data

• Run a Check

• Export Check to wandb

• Export Suite to wandb

Load Data

from deepchecks.tabular.datasets.classification import iris

train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()

classification_label value for label type is deprecated, allowed task types are multiclass, binary and regression.

Run a Check

from deepchecks.tabular.checks import MultivariateDrift

result = MultivariateDrift().add_condition_overall_drift_value_less_than(
).run(train_dataset, test_dataset, model)

Observe CheckResult Display and Value

result

the value can be observed

result.value

{'domain_classifier_auc': 0.4545454545454546, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}

Export a Check’s Output (CheckResult) to wandb

Exporting the output to wandb is possible using the to_wandb function. This function exports the check outputs to a wandb project.

The output display that will be export will be a bit different from what you usually see. Only the tables and the plots are being exported.

On default if you export a single check to wandb without a wandb run active it will create a project with the name deepchecks and the and the check’s metadata in the config and export the results there.

See Check to_wandb options

from deepchecks.core import CheckResult

help(CheckResult.to_wandb)

Help on function to_wandb in module deepchecks.core.check_result:

to_wandb(self, **kwargs)
    Send result to wandb.

    Parameters
    ----------
    kwargs: Keyword arguments to pass to wandb.init.
            Default project name is deepchecks.
            Default config is the check metadata (params, train/test/ name etc.).

To use this function you first need to login to your wandb account

import wandb

wandb.login()

False

result.to_wandb()

wandb: Tracking run with wandb version 0.12.21
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20221031_160319-vnlcmvbc
wandb: Find logs at: ./wandb/offline-run-20221031_160319-vnlcmvbc/logs

Exporting a Suite’s Output (SuiteResult) to wandb

Run Suite and export to wandb

Exporting a suite to wandb is the same as exporting a check. The display in wand will have a section per check.

On default if you export a suite to wandb without a wandb run active it will create a project with the name deepchecks and the suite’s name in the config and send the results there.

from deepchecks.tabular.suites import full_suite

suite = full_suite()

suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.to_wandb()

Full Suite:
|                                   | 0/35 [Time: 00:00]
Full Suite:
|#                                  | 1/35 [Time: 00:00, Check=Train Test Performance]
Full Suite:
|####                               | 4/35 [Time: 00:00, Check=Train Test Prediction Drift]
Full Suite:
|######                             | 6/35 [Time: 00:02, Check=Weak Segments Performance]
Full Suite:
|###########                        | 11/35 [Time: 00:02, Check=Model Inference Time]
Full Suite:
|####################               | 20/35 [Time: 00:02, Check=Feature Label Correlation Change]
Full Suite:
|#########################          | 25/35 [Time: 00:02, Check=Special Characters]
Full Suite:
|#################################  | 33/35 [Time: 00:02, Check=Feature Label Correlation]

wandb: Tracking run with wandb version 0.12.21
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20221031_160327-30n37z4j
wandb: Find logs at: ./wandb/offline-run-20221031_160327-30n37z4j/logs

You can also set all the kwargs the wandb.init will get:

suite_result.to_wandb(project='my-suite-project', config={'suite-name': 'full-suite'})

wandb: Tracking run with wandb version 0.12.21
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20221031_160333-1d5kj07b
wandb: Find logs at: ./wandb/offline-run-20221031_160333-1d5kj07b/logs

previous

Airflow

next

Tabular Checks