Export Outputs to JSON#

In this guide, we will demonstrate how to export the Check’s and the Suite’s output to JSON format.

This enables to use the exported results as a serializeable object that can later be shared or reviewed in other Python scopes, and also enables working with the check’s output (name, parameters, result value and result display) not only in a visual manner (e.g. by parsing the output JSON).

Structure:

Load Data#

from deepchecks.tabular.datasets.classification import iris

train_dataset, test_dataset = iris.load_data()
model = iris.load_fitted_model()

Run a Check#

from deepchecks.tabular.checks import WholeDatasetDrift

result = WholeDatasetDrift().add_condition_overall_drift_value_less_than(
).run(train_dataset, test_dataset, model)

Observe CheckResult Display and Value

result
Whole Dataset Drift


result.value

Out:

{'domain_classifier_auc': 0.4545454545454546, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}

Export a Check’s Output (CheckResult) to a JSON Format#

Serialization of the output to a JSON format is possible using the to_json function. This function takes the check outputs and serializes it to a JSON string.

The format of the check’s output json includes all info required to reconstruct the check run and it’s output: it’s name, the parameters the check receives, one sentence summary of the check’s purpose, it’s result value and a json of the data relevant for displaying the check’s outputs.

See Check JSON Structure#

from deepchecks.core import CheckResult

help(CheckResult.to_json)

Out:

Help on function to_json in module deepchecks.core.check_result:

to_json(self, with_display: bool = True, **kwargs) -> str
    Serialize result into a json string.

    Returned JSON string will have next structure:

    >>    class CheckResultMetadata(TypedDict):
    >>        type: str
    >>        check: CheckMetadata
    >>        value: Any
    >>        header: str
    >>        conditions_results: List[Dict[Any, Any]]
    >>        display: List[Dict[str, Any]]

    >>    class CheckMetadata(TypedDict):
    >>        name: str
    >>        params: Dict[Any, Any]
    >>        summary: str

    Parameters
    ----------
    with_display : bool
        whethere to include display items or not

    Returns
    -------
    str
# get output JSON
serialized_output = result.to_json()
import json

# note - conditions_table key exists only if there are conditions that were defined on check
json.loads(serialized_output).keys()

Out:

dict_keys(['type', 'check', 'header', 'value', 'conditions_results', 'display'])

Observe JSON Output#

json.loads(serialized_output)

Out:

{'type': 'CheckResult', 'check': {'name': 'Whole Dataset Drift', 'params': {'n_top_columns': 3, 'min_feature_importance': 0.05, 'max_num_categories_for_display': 10, 'show_categories_by': 'largest_difference', 'sample_size': 10000, 'random_state': 42, 'test_size': 0.3, 'min_meaningful_drift_score': 0.05}, 'summary': '    Calculate drift between the entire train and test datasets using a model trained to distinguish between them. <a href="https://docs.deepchecks.com/0.8/checks_gallery/tabular/train_test_validation/plot_whole_dataset_drift.html?utm_source=display_output&utm_medium=referral&utm_campaign=check_link" target="_blank">Read More...</a>'}, 'header': 'Whole Dataset Drift', 'value': {'domain_classifier_auc': {'dtype': 'float64', 'value': 0.4545454545454546}, 'domain_classifier_drift_score': 0, 'domain_classifier_feature_importance': {'petal length (cm)': 1.0, 'sepal length (cm)': 0.0, 'sepal width (cm)': 0.0, 'petal width (cm)': 0.0}}, 'conditions_results': [{'Status': 'PASS', 'Condition': 'Drift value is less than 0.25', 'More Info': 'Found drift value of: 0, corresponding to a domain classifier AUC of: 0.45'}], 'display': []}

Display the CheckResult Output from a JSON#

The serialized JSON can be used to reproduce the run in other Python scopes, by using the from_json function

from deepchecks.utils.json_utils import from_json

from_json(serialized_output)
Whole Dataset Drift


Exporting a Suite’s Output (SuiteResult) to JSON#

Run Suite and Save to JSON#

from deepchecks.tabular.suites import full_suite

suite = full_suite()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_json = suite_result.to_json()

Out:

Full Suite:
|                                    | 0/36 [00:00<?, ? Check/s]
Full Suite:
|#                                   | 1/36 [00:00<00:08,  3.99 Check/s, Check=Performance Report]
Full Suite:
|####                                | 4/36 [00:01<00:10,  3.10 Check/s, Check=Segment Performance]
Full Suite:
|######                              | 6/36 [00:01<00:06,  4.64 Check/s, Check=Simple Model Comparison]Default parameter min_samples_leaf will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327
Default parameter smoothing will change in version 2.6.See https://github.com/scikit-learn-contrib/category_encoders/issues/327

Full Suite:
|#######                             | 7/36 [00:01<00:06,  4.73 Check/s, Check=Model Error Analysis]
Full Suite:
|#############                       | 13/36 [00:01<00:01, 12.32 Check/s, Check=Model Inference Time]
Full Suite:
|#######################             | 23/36 [00:01<00:00, 25.92 Check/s, Check=Feature Label Correlation Change]
Full Suite:
|############################        | 28/36 [00:02<00:00, 22.21 Check/s, Check=Special Characters]
Full Suite:
|####################################| 36/36 [00:02<00:00, 28.49 Check/s, Check=Feature Label Correlation]

Observe Suite’s JSON Strucutre#

from deepchecks.core import SuiteResult

help(SuiteResult.to_json)

Out:

Help on function to_json in module deepchecks.core.suite:

to_json(self, with_display: bool = True, **kwargs)
    Return check result as json.

    Parameters
    ----------
    with_display : bool, default True
        whether to include serialized `SuiteResult.display` items into
        the output or not

    Returns
    -------
    str

Suite name:

json.loads(suite_json)['name']

Out:

'Full Suite'

Results is an array of CheckResult JSON’s, let’s see how many checks ran in suite:

len(json.loads(suite_json)['results'])

Out:

54

Lets observe a specific check’s output, which is similar to the Check output’s JSON Structure we saw before for each check

json.loads(suite_json)['results'][0]

Out:

{'type': 'CheckResult', 'check': {'name': 'Performance Report', 'params': {'reduce': 'mean'}, 'summary': 'Summarize given scores on a dataset and model. <a href="https://docs.deepchecks.com/0.8/checks_gallery/tabular/model_evaluation/plot_performance_report.html?utm_source=display_output&utm_medium=referral&utm_campaign=check_link" target="_blank">Read More...</a>'}, 'header': 'Performance Report', 'value': '[{"Dataset":"Train","Class":0,"Metric":"F1","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":1,"Metric":"F1","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":2,"Metric":"F1","Value":1.0,"Number of samples":38},{"Dataset":"Train","Class":0,"Metric":"Precision","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":1,"Metric":"Precision","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":2,"Metric":"Precision","Value":1.0,"Number of samples":38},{"Dataset":"Train","Class":0,"Metric":"Recall","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":1,"Metric":"Recall","Value":1.0,"Number of samples":37},{"Dataset":"Train","Class":2,"Metric":"Recall","Value":1.0,"Number of samples":38},{"Dataset":"Test","Class":0,"Metric":"F1","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":1,"Metric":"F1","Value":0.9285714286,"Number of samples":13},{"Dataset":"Test","Class":2,"Metric":"F1","Value":0.9090909091,"Number of samples":12},{"Dataset":"Test","Class":0,"Metric":"Precision","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":1,"Metric":"Precision","Value":0.8666666667,"Number of samples":13},{"Dataset":"Test","Class":2,"Metric":"Precision","Value":1.0,"Number of samples":12},{"Dataset":"Test","Class":0,"Metric":"Recall","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":1,"Metric":"Recall","Value":1.0,"Number of samples":13},{"Dataset":"Test","Class":2,"Metric":"Recall","Value":0.8333333333,"Number of samples":12}]', 'conditions_results': [{'Status': 'FAIL', 'Condition': 'Train-Test scores relative degradation is not greater than 0.1', 'More Info': 'Precision for class 1 (train=1 test=0.87)\nRecall for class 2 (train=1 test=0.83)'}], 'display': [{'type': 'plotly', 'payload': '{"data":[{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Train<br>Metric=F1<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Train","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"Train","offsetgroup":"Train","orientation":"v","showlegend":true,"x":[0,1,2],"xaxis":"x","y":[1.0,1.0,1.0],"yaxis":"y","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Train<br>Metric=Precision<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Train","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"Train","offsetgroup":"Train","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x2","y":[1.0,1.0,1.0],"yaxis":"y2","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Train<br>Metric=Recall<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Train","marker":{"color":"#636efa","pattern":{"shape":""}},"name":"Train","offsetgroup":"Train","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x3","y":[1.0,1.0,1.0],"yaxis":"y3","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Test<br>Metric=F1<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Test","marker":{"color":"#EF553B","pattern":{"shape":""}},"name":"Test","offsetgroup":"Test","orientation":"v","showlegend":true,"x":[0,1,2],"xaxis":"x","y":[1.0,0.9285714285714286,0.9090909090909091],"yaxis":"y","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Test<br>Metric=Precision<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Test","marker":{"color":"#EF553B","pattern":{"shape":""}},"name":"Test","offsetgroup":"Test","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x2","y":[1.0,0.8666666666666667,1.0],"yaxis":"y2","type":"histogram"},{"alignmentgroup":"True","bingroup":"x","histfunc":"sum","hovertemplate":"Dataset=Test<br>Metric=Recall<br>Class=%{x}<br>sum of Value=%{y}<extra></extra>","legendgroup":"Test","marker":{"color":"#EF553B","pattern":{"shape":""}},"name":"Test","offsetgroup":"Test","orientation":"v","showlegend":false,"x":[0,1,2],"xaxis":"x3","y":[1.0,1.0,0.8333333333333334],"yaxis":"y3","type":"histogram"}],"layout":{"template":{"data":{"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"#E5ECF6","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"barpolar":[{"marker":{"line":{"color":"#E5ECF6","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"white","linecolor":"white","minorgridcolor":"white","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"white","linecolor":"white","minorgridcolor":"white","startlinecolor":"#2a3f5f"},"type":"carpet"}],"choropleth":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"choropleth"}],"contour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"contour"}],"contourcarpet":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"contourcarpet"}],"heatmap":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmap"}],"heatmapgl":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmapgl"}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"histogram2d":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2d"}],"histogram2dcontour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2dcontour"}],"mesh3d":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"mesh3d"}],"parcoords":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"parcoords"}],"pie":[{"automargin":true,"type":"pie"}],"scatter":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter"}],"scatter3d":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter3d"}],"scattercarpet":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattercarpet"}],"scattergeo":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergeo"}],"scattergl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergl"}],"scattermapbox":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermapbox"}],"scatterpolar":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolar"}],"scatterpolargl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolargl"}],"scatterternary":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterternary"}],"surface":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"surface"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}]},"layout":{"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"autotypenumbers":"strict","coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]],"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]},"colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"geo":{"bgcolor":"white","lakecolor":"white","landcolor":"#E5ECF6","showlakes":true,"showland":true,"subunitcolor":"white"},"hoverlabel":{"align":"left"},"hovermode":"closest","mapbox":{"style":"light"},"paper_bgcolor":"white","plot_bgcolor":"#E5ECF6","polar":{"angularaxis":{"gridcolor":"white","linecolor":"white","ticks":""},"bgcolor":"#E5ECF6","radialaxis":{"gridcolor":"white","linecolor":"white","ticks":""}},"scene":{"xaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","gridwidth":2,"linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white"},"yaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","gridwidth":2,"linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white"},"zaxis":{"backgroundcolor":"#E5ECF6","gridcolor":"white","gridwidth":2,"linecolor":"white","showbackground":true,"ticks":"","zerolinecolor":"white"}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"ternary":{"aaxis":{"gridcolor":"white","linecolor":"white","ticks":""},"baxis":{"gridcolor":"white","linecolor":"white","ticks":""},"bgcolor":"#E5ECF6","caxis":{"gridcolor":"white","linecolor":"white","ticks":""}},"title":{"x":0.05},"xaxis":{"automargin":true,"gridcolor":"white","linecolor":"white","ticks":"","title":{"standoff":15},"zerolinecolor":"white","zerolinewidth":2},"yaxis":{"automargin":true,"gridcolor":"white","linecolor":"white","ticks":"","title":{"standoff":15},"zerolinecolor":"white","zerolinewidth":2}}},"xaxis":{"anchor":"y","domain":[0.0,0.3],"title":{},"tickprefix":"Class ","tickangle":60,"type":"category"},"yaxis":{"anchor":"x","domain":[0.0,1.0],"title":{},"showticklabels":true},"xaxis2":{"anchor":"y2","domain":[0.35,0.6499999999999999],"matches":"x","title":{},"tickprefix":"Class ","tickangle":60,"type":"category"},"yaxis2":{"anchor":"x2","domain":[0.0,1.0],"showticklabels":true,"title":{}},"xaxis3":{"anchor":"y3","domain":[0.7,1.0],"matches":"x","title":{},"tickprefix":"Class ","tickangle":60,"type":"category"},"yaxis3":{"anchor":"x3","domain":[0.0,1.0],"showticklabels":true,"title":{}},"annotations":[{"font":{},"showarrow":false,"text":"F1","x":0.15,"xanchor":"center","xref":"paper","y":1.0,"yanchor":"bottom","yref":"paper"},{"font":{},"showarrow":false,"text":"Precision","x":0.49999999999999994,"xanchor":"center","xref":"paper","y":1.0,"yanchor":"bottom","yref":"paper"},{"font":{},"showarrow":false,"text":"Recall","x":0.85,"xanchor":"center","xref":"paper","y":1.0,"yanchor":"bottom","yref":"paper"}],"legend":{"title":{"text":"Dataset"},"tracegroupgap":0},"margin":{"t":60},"barmode":"group"}}'}]}

Total running time of the script: ( 0 minutes 4.739 seconds)

Gallery generated by Sphinx-Gallery