Note
Click here to download the full example code
Export Outputs to Weights & Biases (wandb)#
In this guide, we will demonstrate how to export the Check’s and the Suite’s output to wandb.
This enables to view the exported results in wandb and compare the results you receive with different parameters or model/data.
Structure:
Load Data#
from deepchecks.tabular.datasets.classification import iris
train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()
Run a Check#
from deepchecks.tabular.checks import WholeDatasetDrift
result = WholeDatasetDrift().add_condition_overall_drift_value_not_greater_than(
).run(train_dataset, test_dataset, model)
Out:
Calculating permutation feature importance. Expected to finish in 1 seconds
Observe CheckResult Display and Value#
result
the value can be observed
result.value
Out:
{'domain_classifier_auc': 0.4545454545454546, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}
Export a Check’s Output (CheckResult) to wandb#
Exporting the output to wandb is possible using the to_wandb function. This function
exports the check outputs to a wandb project.
The output display that will be export will be a bit different from what you usually see. Only the tables and the plots are being exported.
On default if you export a single check to wandb without a wandb run active it will create a project with the name deepchecks and the and the check’s metadata in the config and export the results there.
See Check to_wandb options#
from deepchecks.core import CheckResult
help(CheckResult.to_wandb)
Out:
Help on function to_wandb in module deepchecks.core.check_result:
to_wandb(self, dedicated_run: bool = True, **kwargs: Any)
Export check result to wandb.
Parameters
----------
dedicated_run : bool , default: None
If to initiate and finish a new wandb run.
If None it will be dedicated if wandb.run is None.
kwargs: Keyword arguments to pass to wandb.init.
Default project name is deepchecks.
Default config is the check metadata (params, train/test/ name etc.).
To use this function you first need to login to your wandb account
import wandb
wandb.login()
Out:
False
result.to_wandb()
Out:
wandb: Tracking run with wandb version 0.12.16
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20220512_143641-2mm5gmi8
wandb: Find logs at: ./wandb/offline-run-20220512_143641-2mm5gmi8/logs
Exporting a Suite’s Output (SuiteResult) to wandb#
Run Suite and export to wandb#
Exporting a suite to wandb is the same as exporting a check. The display in wand will have a section per check.
On default if you export a suite to wandb without a wandb run active it will create a project with the name deepchecks and the suite’s name in the config and send the results there.
from deepchecks.tabular.suites import full_suite
suite = full_suite()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.to_wandb()
Out:
Full Suite: 0%| | 0/36 [00:00<?, ? Check/s]
Full Suite: 0%| | 0/36 [00:00<?, ? Check/s, Check=Model Info]
Full Suite: 3%|# | 1/36 [00:00<00:00, 52.23 Check/s, Check=Columns Info]
Full Suite: 6%|## | 2/36 [00:00<00:00, 35.12 Check/s, Check=Confusion Matrix Report]
Full Suite: 8%|### | 3/36 [00:00<00:00, 33.40 Check/s, Check=Performance Report]
Full Suite: 11%|#### | 4/36 [00:00<00:02, 14.50 Check/s, Check=Performance Report]
Full Suite: 11%|#### | 4/36 [00:00<00:02, 14.50 Check/s, Check=Roc Report]
Full Suite: 14%|##### | 5/36 [00:00<00:02, 14.50 Check/s, Check=Simple Model Comparison]
Full Suite: 17%|###### | 6/36 [00:00<00:01, 15.91 Check/s, Check=Simple Model Comparison]
Full Suite: 17%|###### | 6/36 [00:00<00:01, 15.91 Check/s, Check=Model Error Analysis]
Full Suite: 19%|####### | 7/36 [00:00<00:01, 15.91 Check/s, Check=Calibration Score]
Full Suite: 22%|######## | 8/36 [00:00<00:02, 13.36 Check/s, Check=Calibration Score]
Full Suite: 22%|######## | 8/36 [00:00<00:02, 13.36 Check/s, Check=Regression Systematic Error]
Full Suite: 25%|######### | 9/36 [00:00<00:02, 13.36 Check/s, Check=Regression Error Distribution]
Full Suite: 28%|########## | 10/36 [00:00<00:01, 13.36 Check/s, Check=Boosting Overfit]
Full Suite: 31%|########### | 11/36 [00:00<00:01, 13.36 Check/s, Check=Unused Features]
Full Suite: 33%|############ | 12/36 [00:00<00:01, 13.36 Check/s, Check=Model Inference Time]
Full Suite: 36%|############# | 13/36 [00:00<00:01, 13.36 Check/s, Check=Train Test Feature Drift]
Full Suite: 39%|############## | 14/36 [00:00<00:01, 21.30 Check/s, Check=Train Test Feature Drift]
Full Suite: 39%|############## | 14/36 [00:00<00:01, 21.30 Check/s, Check=Train Test Label Drift]
Full Suite: 42%|############### | 15/36 [00:00<00:00, 21.30 Check/s, Check=Whole Dataset Drift] Calculating permutation feature importance. Expected to finish in 1 seconds
Full Suite: 44%|################ | 16/36 [00:00<00:00, 21.30 Check/s, Check=Dominant Frequency Change]
Full Suite: 47%|################# | 17/36 [00:00<00:00, 21.30 Check/s, Check=Category Mismatch Train Test]
Full Suite: 50%|################## | 18/36 [00:00<00:00, 21.30 Check/s, Check=New Label Train Test]
Full Suite: 53%|################### | 19/36 [00:00<00:00, 21.30 Check/s, Check=String Mismatch Comparison]
Full Suite: 56%|#################### | 20/36 [00:00<00:00, 30.36 Check/s, Check=String Mismatch Comparison]
Full Suite: 56%|#################### | 20/36 [00:00<00:00, 30.36 Check/s, Check=Datasets Size Comparison]
Full Suite: 58%|##################### | 21/36 [00:00<00:00, 30.36 Check/s, Check=Date Train Test Leakage Duplicates]
Full Suite: 61%|###################### | 22/36 [00:00<00:00, 30.36 Check/s, Check=Date Train Test Leakage Overlap]
Full Suite: 64%|####################### | 23/36 [00:00<00:00, 30.36 Check/s, Check=Single Feature Contribution Train Test]
Full Suite: 67%|######################## | 24/36 [00:00<00:00, 30.36 Check/s, Check=Train Test Samples Mix]
Full Suite: 69%|######################### | 25/36 [00:00<00:00, 30.36 Check/s, Check=Identifier Leakage]
Full Suite: 72%|########################## | 26/36 [00:00<00:00, 30.36 Check/s, Check=Index Train Test Leakage]
Full Suite: 75%|########################### | 27/36 [00:00<00:00, 30.36 Check/s, Check=Is Single Value]
Full Suite: 78%|############################ | 28/36 [00:00<00:00, 30.36 Check/s, Check=Mixed Nulls]
Full Suite: 81%|############################# | 29/36 [00:00<00:00, 30.36 Check/s, Check=Mixed Data Types]
Full Suite: 83%|############################## | 30/36 [00:00<00:00, 30.36 Check/s, Check=String Mismatch]
Full Suite: 86%|############################### | 31/36 [00:00<00:00, 30.36 Check/s, Check=Data Duplicates]
Full Suite: 89%|################################ | 32/36 [00:00<00:00, 30.36 Check/s, Check=String Length Out Of Bounds]
Full Suite: 92%|################################# | 33/36 [00:00<00:00, 30.36 Check/s, Check=Special Characters]
Full Suite: 94%|################################## | 34/36 [00:00<00:00, 58.00 Check/s, Check=Special Characters]
Full Suite: 94%|################################## | 34/36 [00:00<00:00, 58.00 Check/s, Check=Conflicting Labels]
Full Suite: 97%|################################### | 35/36 [00:00<00:00, 58.00 Check/s, Check=Outlier Sample Detection]
wandb: Tracking run with wandb version 0.12.16
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20220512_143646-owxq9nf8
wandb: Find logs at: ./wandb/offline-run-20220512_143646-owxq9nf8/logs
You can also set all the kwargs the wandb.init will get:
suite_result.to_wandb(project='my-suite-project', config={'suite-name': 'full-suite'})
Out:
wandb: Tracking run with wandb version 0.12.16
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: \ 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/runner/work/deepchecks/deepchecks/docs/source/user-guide/general/exporting_results/wandb/offline-run-20220512_143651-2dtzth2j
wandb: Find logs at: ./wandb/offline-run-20220512_143651-2dtzth2j/logs
Total running time of the script: ( 0 minutes 19.176 seconds)