Note
Click here to download the full example code
Export Outputs to JSON#
In this guide, we will demonstrate how to export the Check’s and the Suite’s output to JSON format.
This enables to use the exported results as a serializeable object that can later be shared or reviewed in other Python scopes, and also enables working with the check’s output (name, parameters, result value and result display) not only in a visual manner (e.g. by parsing the output JSON).
Structure:
Load Data#
from deepchecks.tabular.datasets.classification import iris
train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()
Run a Check#
from deepchecks.tabular.checks import WholeDatasetDrift
result = WholeDatasetDrift().add_condition_overall_drift_value_less_than(
).run(train_dataset, test_dataset, model)
Observe CheckResult Display and Value
result
result.value
Out:
{'domain_classifier_auc': 0.4545454545454546, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}
Export a Check’s Output (CheckResult) to a JSON Format#
Serialization of the output to a JSON format is possible using the to_json function.
This function takes the check outputs and serializes it to a JSON string.
The format of the check’s output json includes all info required to reconstruct the check run and it’s output: it’s name, the parameters the check receives, one sentence summary of the check’s purpose, it’s result value and a json of the data relevant for displaying the check’s outputs.
See Check JSON Structure#
from deepchecks.core import CheckResult
help(CheckResult.to_json)
Out:
Help on function to_json in module deepchecks.core.check_result:
to_json(self, with_display: bool = True, **kwargs) -> str
Serialize result into a json string.
Returned JSON string will have next structure:
>> class CheckResultMetadata(TypedDict):
>> type: str
>> check: CheckMetadata
>> value: Any
>> header: str
>> conditions_results: List[Dict[Any, Any]]
>> display: List[Dict[str, Any]]
>> class CheckMetadata(TypedDict):
>> name: str
>> params: Dict[Any, Any]
>> summary: str
Parameters
----------
with_display : bool
whethere to include display items or not
Returns
-------
str
# get output JSON
serialized_output = result.to_json()
import json
# note - conditions_table key exists only if there are conditions that were defined on check
json.loads(serialized_output).keys()
Out:
dict_keys(['type', 'check', 'header', 'value', 'conditions_results', 'display'])
Observe JSON Output#
json.loads(serialized_output)
Out:
{'type': 'CheckResult', 'check': {'name': 'Whole Dataset Drift', 'params': {'n_top_columns': 3, 'min_feature_importance': 0.05, 'max_num_categories_for_display': 10, 'show_categories_by': 'largest_difference', 'sample_size': 10000, 'random_state': 42, 'test_size': 0.3, 'min_meaningful_drift_score': 0.05}, 'summary': ' Calculate drift between the entire train and test datasets using a model trained to distinguish between them. <a href="https://docs.deepchecks.com/0.8/checks_gallery/tabular/train_test_validation/plot_whole_dataset_drift.html?utm_source=display_output&utm_medium=referral&utm_campaign=check_link" target="_blank">Read More...</a>'}, 'header': 'Whole Dataset Drift', 'value': {'domain_classifier_auc': {'dtype': 'float64', 'value': 0.4545454545454546}, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}, 'conditions_results': [{'Status': 'PASS', 'Condition': 'Drift value is less than 0.25', 'More Info': 'Found drift value of: 0, corresponding to a domain classifier AUC of: 0.45'}], 'display': []}
Display the CheckResult Output from a JSON#
The serialized JSON can be used to reproduce the run in other Python scopes,
by using the from_json function
from deepchecks.utils.json_utils import from_json
from_json(serialized_output)
Exporting a Suite’s Output (SuiteResult) to JSON#
Run Suite and Save to JSON#
from deepchecks.tabular.suites import full_suite
suite = full_suite()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_json = suite_result.to_json()
Out:
Full Suite:
| | 0/36 [00:00<?, ? Check/s]
Full Suite:
|# | 1/36 [00:00<00:08, 3.99 Check/s, Check=Performance Report]
Full Suite:
|#### | 4/36 [00:01<00:10, 3.10 Check/s, Check=Segment Performance]
Full Suite:
|###### | 6/36 [00:01<00:06, 4.64 Check/s, Check=Simple Model Comparison]Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Full Suite:
|####### | 7/36 [00:01<00:06, 4.73 Check/s, Check=Model Error Analysis]
Full Suite:
|############# | 13/36 [00:01<00:01, 12.32 Check/s, Check=Model Inference Time]
Full Suite:
|####################### | 23/36 [00:01<00:00, 25.92 Check/s, Check=Feature Label Correlation Change]
Full Suite:
|############################ | 28/36 [00:02<00:00, 22.21 Check/s, Check=Special Characters]
Full Suite:
|####################################| 36/36 [00:02<00:00, 28.49 Check/s, Check=Feature Label Correlation]
Observe Suite’s JSON Strucutre#
from deepchecks.core import SuiteResult
help(SuiteResult.to_json)
Out:
Help on function to_json in module deepchecks.core.suite:
to_json(self, with_display: bool = True, **kwargs)
Return check result as json.
Parameters
----------
with_display : bool, default True
whether to include serialized `SuiteResult.display` items into
the output or not
Returns
-------
str
Suite name:
json.loads(suite_json)['name']
Out:
'Full Suite'
Results is an array of CheckResult JSON’s, let’s see how many checks ran in suite:
len(json.loads(suite_json)['results'])
Out:
54
Lets observe a specific check’s output, which is similar to the Check output’s JSON Structure we saw before for each check
json.loads(suite_json)['results'][0]
Out:
{'type': 'CheckResult', 'check': {'name': 'Performance Report', 'params': {'reduce': 'mean'}, 'summary': 'Summarize given scores on a dataset and model. <a href="https://docs.deepchecks.com/0.8/checks_gallery/tabular/model_evaluation/plot_performance_report.html?utm_source=display_output&utm_medium=referral&utm_campaign=check_link" target="_blank">Read More...</a>'}, 'header': 'Performance Report', 'value': '[{"Dataset":"Train","Class":0,"Metric":"F1","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":1,"Metric":"F1","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":2,"Metric":"F1","Value":1.0,"Number of samples":38},{"Dataset":"Train","Class":0,"Metric":"Precision","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":1,"Metric":"Precision","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":2,"Metric":"Precision","Value":1.0,"Number of samples":38},{"Dataset":"Train","Class":0,"Metric":"Recall","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":1,"Metric":"Recall","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":2,"Metric":"Recall","Value":1.0,"Number of samples":38},{"Dataset":"Test","Class":0,"Metric":"F1","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":1,"Metric":"F1","Value":0.9285714286,"Number of samples":13},{"Dataset":"Test","Class":2,"Metric":"F1","Value":0.9090909091,"Number of samples":12},{"Dataset":"Test","Class":0,"Metric":"Precision","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":1,"Metric":"Precision","Value":0.8666666667,"Number of samples":13},{"Dataset":"Test","Class":2,"Metric":"Precision","Value":1.0,"Number of samples":12},{"Dataset":"Test","Class":0,"Metric":"Recall","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":1,"Metric":"Recall","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":2,"Metric":"Recall","Value":0.8333333333,"Number of samples":12}]', 'conditions_results': [{'Status': 'FAIL', 'Condition': 'Train-Test scores relative degradation is not greater than 0.1', 'More Info': 'Precision for class 1 (train=1 test=0.87)\nRecall for class 2 (train=1 test=0.83)'}], 'display': [{'type': 'plotly', 'payload': '{"data":[{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Train<br>Metric=F1<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Train","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"Train","offsetgroup":"Train","orientation":"v","showlegend":true,"x":[0,1,2],"xaxis":"x","y":[1.0,1.0,1.0],"yaxis":"y","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Train<br>Metric=Precision<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Train","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"Train","offsetgroup":"Train","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x2","y":[1.0,1.0,1.0],"yaxis":"y2","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Train<br>Metric=Recall<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Train","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"Train","offsetgroup":"Train","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x3","y":[1.0,1.0,1.0],"yaxis":"y3","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Test<br>Metric=F1<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Test","marker":{"color":"#EF553B","pattern":{"shape":""}},"name":"Test","offsetgroup":"Test","orientation":"v","showlegend":true,"x":[0,1,2],"xaxis":"x","y":[1.0,0.9285714285714286,0.9090909090909091],"yaxis":"y","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Test<br>Metric=Precision<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Test","marker":{"color":"#EF553B","pattern":{"shape":""}},"name":"Test","offsetgroup":"Test","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x2","y":[1.0,0.8666666666666667,1.0],"yaxis":"y2","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Test<br>Metric=Recall<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Test","marker":{"color":"#EF553B","pattern":{"shape":""}},"name":"Test","offsetgroup":"Test","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x3","y":[1.0,1.0,0.8333333333333334],"yaxis":"y3","type":"histogram"}],"layout":{"template":{"data":{"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"#E5ECF6","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"barpolar":[{"marker":{"line":{"color":"#E5ECF6","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"white","linecolor":"white","minorgridcolor":"white","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"white","linecolor":"white","minorgridcolor":"white","startlinecolor":"#2a3f5f"},"type":"carpet"}],"choropleth":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"choropleth"}],"contour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"contour"}],"contourcarpet":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"contourcarpet"}],"heatmap":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmap"}],"heatmapgl":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmapgl"}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"histogram2d":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2d"}],"histogram2dcontour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2dcontour"}],"mesh3d":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"mesh3d"}],"parcoords":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"parcoords"}],"pie":[{"automargin":true,"type":"pie"}],"scatter":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter"}],"scatter3d":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter3d"}],"scattercarpet":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattercarpet"}],"scattergeo":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergeo"}],"scattergl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergl"}],"scattermapbox":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermapbox"}],"scatterpolar":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolar"}],"scatterpolargl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolargl"}],"scatterternary":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterternary"}],"surface":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"surface"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}]},"layout":{"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"autotypenumbers":"strict","coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]],"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]},"colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"geo":{"bgcolor":"white","lakecolor":"white","landcolor":"#E5ECF6","showlakes":true,"showland":true,"subunitcolor":"white"},"hoverlabel":{"align":"left"},"hovermode":"closest","mapbox":{"style":"light"},"paper_bgcolor":"white","plot_bgcolor":"#E5ECF6","polar":{"angularaxis":{"gridcolor":"white","linecolor":"white","ticks":""},"bgcolor":"#E5ECF6","radialaxis":{"gridcolor":"white","linecolor":"white","ticks":""}},"scene":{"xaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","gridwidth":2,"linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white"},"yaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","gridwidth":2,"linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white"},"zaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","gridwidth":2,"linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white"}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"ternary":{"aaxis":{"gridcolor":"white","linecolor":"white","ticks":""},"baxis":{"gridcolor":"white","linecolor":"white","ticks":""},"bgcolor":"#E5ECF6","caxis":{"gridcolor":"white","linecolor":"white","ticks":""}},"title":{"x":0.05},"xaxis":{"automargin":true,"gridcolor":"white","linecolor":"white","ticks":"","title":{"standoff":15},"zerolinecolor":"white","zerolinewidth":2},"yaxis":{"automargin":true,"gridcolor":"white","linecolor":"white","ticks":"","title":{"standoff":15},"zerolinecolor":"white","zerolinewidth":2}}},"xaxis":{"anchor":"y","domain":[0.0,0.3],"title":{},"tickprefix":"Class ","tickangle":60,"type":"category"},"yaxis":{"anchor":"x","domain":[0.0,1.0],"title":{},"showticklabels":true},"xaxis2":{"anchor":"y2","domain":[0.35,0.6499999999999999],"matches":"x","title":{},"tickprefix":"Class ","tickangle":60,"type":"category"},"yaxis2":{"anchor":"x2","domain":[0.0,1.0],"showticklabels":true,"title":{}},"xaxis3":{"anchor":"y3","domain":[0.7,1.0],"matches":"x","title":{},"tickprefix":"Class ","tickangle":60,"type":"category"},"yaxis3":{"anchor":"x3","domain":[0.0,1.0],"showticklabels":true,"title":{}},"annotations":[{"font":{},"showarrow":false,"text":"F1","x":0.15,"xanchor":"center","xref":"paper","y":1.0,"yanchor":"bottom","yref":"paper"},{"font":{},"showarrow":false,"text":"Precision","x":0.49999999999999994,"xanchor":"center","xref":"paper","y":1.0,"yanchor":"bottom","yref":"paper"},{"font":{},"showarrow":false,"text":"Recall","x":0.85,"xanchor":"center","xref":"paper","y":1.0,"yanchor":"bottom","yref":"paper"}],"legend":{"title":{"text":"Dataset"},"tracegroupgap":0},"margin":{"t":60},"barmode":"group"}}'}]}
Total running time of the script: ( 0 minutes 4.739 seconds)