Note
Click here to download the full example code
Export Outputs to JSON#
In this guide, we will demonstrate how to export the Check’s and the Suite’s output to JSON format.
This enables to use the exported results as a serializeable object that can later be shared or reviewed in other Python scopes, and also enables working with the check’s output (name, parameters, result value and result display) not only in a visual manner (e.g. by parsing the output JSON).
Structure:
Load Data#
from deepchecks.tabular.datasets.classification import iris
train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()
Run a Check#
from deepchecks.tabular.checks import WholeDatasetDrift
result = WholeDatasetDrift().add_condition_overall_drift_value_not_greater_than(
).run(train_dataset, test_dataset, model)
Out:
Calculating permutation feature importance. Expected to finish in 1 seconds
Observe CheckResult Display and Value
result
result.value
Out:
{'domain_classifier_auc': 0.4545454545454546, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}
Export a Check’s Output (CheckResult) to a JSON Format#
Serialization of the output to a JSON format is possible using the to_json function.
This function takes the check outputs and serializes it to a JSON string.
The format of the check’s output json includes all info required to reconstruct the check run and it’s output: it’s name, the parameters the check receives, one sentence summary of the check’s purpose, it’s result value and a json of the data relevant for displaying the check’s outputs.
See Check JSON Structure#
from deepchecks.core import CheckResult
help(CheckResult.to_json)
Out:
Help on function to_json in module deepchecks.core.check_result:
to_json(self, with_display: bool = True) -> str
Return check result as json.
Returned JSON string will have next structure:
>> class CheckResultMetadata(TypedDict):
>> type: str
>> check: CheckMetadata
>> value: Any
>> header: str
>> conditions_results: List[Dict[Any, Any]]
>> display: List[Dict[str, Any]]
>> class CheckMetadata(TypedDict):
>> name: str
>> params: Dict[Any, Any]
>> summary: str
Parameters
----------
with_display : bool
controls if to serialize display or not
Returns
-------
str
# get output JSON
serialized_output = result.to_json()
import json
# note - conditions_table key exists only if there are conditions that were defined on check
json.loads(serialized_output).keys()
Out:
dict_keys(['type', 'check', 'header', 'value', 'conditions_results', 'display'])
Observe JSON Output#
json.loads(serialized_output)
Out:
{'type': 'CheckResult', 'check': {'name': 'Whole Dataset Drift', 'params': {'n_top_columns': 3, 'min_feature_importance': 0.05, 'max_num_categories_for_display': 10, 'show_categories_by': 'largest_difference', 'sample_size': 10000, 'random_state': 42, 'test_size': 0.3, 'min_meaningful_drift_score': 0.05}, 'summary': ' Calculate drift between the entire train and test datasets using a model trained to distinguish between them. <a href="https://docs.deepchecks.com/0.6/checks_gallery/tabular/distribution/plot_whole_dataset_drift.html?utm_source=display_output&utm_medium=referral&utm_campaign=check_link" target="_blank">Read More...</a>'}, 'header': 'Whole Dataset Drift', 'value': {'domain_classifier_auc': {'dtype': 'float64', 'value': 0.4545454545454546}, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}, 'conditions_results': [{'Status': 'PASS', 'Condition': 'Drift value is not greater than 0.25', 'More Info': ''}], 'display': []}
Display the CheckResult Output from a JSON#
The serialized JSON can be used to reproduce the run in other Python scopes,
by using the from_json function
from deepchecks.utils.json_utils import from_json
from_json(serialized_output)
Exporting a Suite’s Output (SuiteResult) to JSON#
Run Suite and Save to JSON#
from deepchecks.tabular.suites import full_suite
suite = full_suite()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_json = suite_result.to_json()
Out:
Full Suite: 0%| | 0/36 [00:00<?, ? Check/s]
Full Suite: 0%| | 0/36 [00:00<?, ? Check/s, Check=Model Info]
Full Suite: 3%|# | 1/36 [00:00<00:00, 61.55 Check/s, Check=Columns Info]
Full Suite: 6%|## | 2/36 [00:00<00:00, 35.87 Check/s, Check=Confusion Matrix Report]
Full Suite: 8%|### | 3/36 [00:00<00:01, 32.57 Check/s, Check=Performance Report]
Full Suite: 11%|#### | 4/36 [00:00<00:02, 13.71 Check/s, Check=Performance Report]
Full Suite: 11%|#### | 4/36 [00:00<00:02, 13.71 Check/s, Check=Roc Report]
Full Suite: 14%|##### | 5/36 [00:00<00:02, 13.71 Check/s, Check=Simple Model Comparison]
Full Suite: 17%|###### | 6/36 [00:00<00:02, 14.78 Check/s, Check=Simple Model Comparison]
Full Suite: 17%|###### | 6/36 [00:00<00:02, 14.78 Check/s, Check=Model Error Analysis]
Full Suite: 19%|####### | 7/36 [00:00<00:01, 14.78 Check/s, Check=Calibration Score]
Full Suite: 22%|######## | 8/36 [00:00<00:02, 12.03 Check/s, Check=Calibration Score]
Full Suite: 22%|######## | 8/36 [00:00<00:02, 12.03 Check/s, Check=Regression Systematic Error]
Full Suite: 25%|######### | 9/36 [00:00<00:02, 12.03 Check/s, Check=Regression Error Distribution]
Full Suite: 28%|########## | 10/36 [00:00<00:02, 12.03 Check/s, Check=Boosting Overfit]
Full Suite: 31%|########### | 11/36 [00:00<00:02, 12.03 Check/s, Check=Unused Features]
Full Suite: 33%|############ | 12/36 [00:00<00:01, 12.03 Check/s, Check=Model Inference Time]
Full Suite: 36%|############# | 13/36 [00:00<00:01, 12.03 Check/s, Check=Train Test Feature Drift]
Full Suite: 39%|############## | 14/36 [00:00<00:01, 18.59 Check/s, Check=Train Test Feature Drift]
Full Suite: 39%|############## | 14/36 [00:00<00:01, 18.59 Check/s, Check=Train Test Label Drift]
Full Suite: 42%|############### | 15/36 [00:00<00:01, 18.59 Check/s, Check=Whole Dataset Drift] Calculating permutation feature importance. Expected to finish in 1 seconds
Full Suite: 44%|################ | 16/36 [00:00<00:01, 18.54 Check/s, Check=Whole Dataset Drift]
Full Suite: 44%|################ | 16/36 [00:00<00:01, 18.54 Check/s, Check=Dominant Frequency Change]
Full Suite: 47%|################# | 17/36 [00:00<00:01, 18.54 Check/s, Check=Category Mismatch Train Test]
Full Suite: 50%|################## | 18/36 [00:00<00:00, 18.54 Check/s, Check=New Label Train Test]
Full Suite: 53%|################### | 19/36 [00:00<00:00, 18.54 Check/s, Check=String Mismatch Comparison]
Full Suite: 56%|#################### | 20/36 [00:00<00:00, 18.54 Check/s, Check=Datasets Size Comparison]
Full Suite: 58%|##################### | 21/36 [00:00<00:00, 18.54 Check/s, Check=Date Train Test Leakage Duplicates]
Full Suite: 61%|###################### | 22/36 [00:00<00:00, 18.54 Check/s, Check=Date Train Test Leakage Overlap]
Full Suite: 64%|####################### | 23/36 [00:00<00:00, 18.54 Check/s, Check=Single Feature Contribution Train Test]
Full Suite: 67%|######################## | 24/36 [00:01<00:00, 18.54 Check/s, Check=Train Test Samples Mix]
Full Suite: 69%|######################### | 25/36 [00:01<00:00, 18.54 Check/s, Check=Identifier Leakage]
Full Suite: 72%|########################## | 26/36 [00:01<00:00, 37.32 Check/s, Check=Identifier Leakage]
Full Suite: 72%|########################## | 26/36 [00:01<00:00, 37.32 Check/s, Check=Index Train Test Leakage]
Full Suite: 75%|########################### | 27/36 [00:01<00:00, 37.32 Check/s, Check=Is Single Value]
Full Suite: 78%|############################ | 28/36 [00:01<00:00, 37.32 Check/s, Check=Mixed Nulls]
Full Suite: 81%|############################# | 29/36 [00:01<00:00, 37.32 Check/s, Check=Mixed Data Types]
Full Suite: 83%|############################## | 30/36 [00:01<00:00, 37.32 Check/s, Check=String Mismatch]
Full Suite: 86%|############################### | 31/36 [00:01<00:00, 37.32 Check/s, Check=Data Duplicates]
Full Suite: 89%|################################ | 32/36 [00:01<00:00, 37.32 Check/s, Check=String Length Out Of Bounds]
Full Suite: 92%|################################# | 33/36 [00:01<00:00, 37.32 Check/s, Check=Special Characters]
Full Suite: 94%|################################## | 34/36 [00:01<00:00, 37.32 Check/s, Check=Conflicting Labels]
Full Suite: 97%|################################### | 35/36 [00:01<00:00, 37.32 Check/s, Check=Outlier Sample Detection]
Observe Suite’s JSON Strucutre#
from deepchecks.core import SuiteResult
help(SuiteResult.to_json)
Out:
Help on function to_json in module deepchecks.core.suite:
to_json(self, with_display: bool = True)
Return check result as json.
Parameters
----------
with_display : bool
controls if to serialize display of checks or not
Returns
-------
str
Suite name:
json.loads(suite_json)['name']
Out:
'Full Suite'
Results is an array of CheckResult JSON’s, let’s see how many checks ran in suite:
len(json.loads(suite_json)['results'])
Out:
53
Lets observe a specific check’s output, which is similar to the Check output’s JSON Structure we saw before for each check
json.loads(suite_json)['results'][0]
Out:
{'type': 'CheckResult', 'check': {'name': 'Model Info', 'params': {}, 'summary': 'Summarize given model parameters. <a href="https://docs.deepchecks.com/0.6/checks_gallery/tabular/overview/plot_model_info.html?utm_source=display_output&utm_medium=referral&utm_campaign=check_link" target="_blank">Read More...</a>'}, 'header': 'Model Info', 'value': {'type': 'RandomForestClassifier', 'params': {'bootstrap': True, 'ccp_alpha': 0.0, 'class_weight': None, 'criterion': 'gini', 'max_depth': None, 'max_features': 'auto', 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 100, 'n_jobs': None, 'oob_score': False, 'random_state': None, 'verbose': 0, 'warm_start': False}}, 'conditions_results': [], 'display': [{'type': 'html', 'payload': 'Model Type: RandomForestClassifier'}, {'type': 'dataframe', 'payload': [{'Parameter': 'bootstrap', 'Value': True, 'Default': True}, {'Parameter': 'ccp_alpha', 'Value': 0.0, 'Default': 0.0}, {'Parameter': 'class_weight', 'Value': None, 'Default': None}, {'Parameter': 'criterion', 'Value': 'gini', 'Default': 'gini'}, {'Parameter': 'max_depth', 'Value': None, 'Default': None}, {'Parameter': 'max_features', 'Value': 'auto', 'Default': 'auto'}, {'Parameter': 'max_leaf_nodes', 'Value': None, 'Default': None}, {'Parameter': 'max_samples', 'Value': None, 'Default': None}, {'Parameter': 'min_impurity_decrease', 'Value': 0.0, 'Default': 0.0}, {'Parameter': 'min_samples_leaf', 'Value': 1, 'Default': 1}, {'Parameter': 'min_samples_split', 'Value': 2, 'Default': 2}, {'Parameter': 'min_weight_fraction_leaf', 'Value': 0.0, 'Default': 0.0}, {'Parameter': 'n_estimators', 'Value': 100, 'Default': 100}, {'Parameter': 'n_jobs', 'Value': None, 'Default': None}, {'Parameter': 'oob_score', 'Value': False, 'Default': False}, {'Parameter': 'random_state', 'Value': None, 'Default': None}, {'Parameter': 'verbose', 'Value': 0, 'Default': 0}, {'Parameter': 'warm_start', 'Value': False, 'Default': False}]}, {'type': 'html', 'payload': '<p style="font-size:0.7em"><i>Colored rows are parameters with non-default values</i></p>'}]}
Total running time of the script: ( 0 minutes 5.111 seconds)