Model Performance Insights

DataRobot provides several insights to help you understand and evaluate model performance:

ROC Curve: Receiver Operating Characteristic curve showing the trade-off between true positive rate and false positive rate at various classification thresholds. Available for binary classification models.
Lift Chart: Depicts how well a model segments the target population and how well the model performs for different ranges of values of the target variable. Available for classification and regression models.
Residuals: Displays the distribution of prediction errors for regression models.

These insights can be computed for different data partitions (validation, cross-validation, holdout) and can be filtered by data slices.

The following example code assumes that you have a trained model object called model.

ROC Curve

The following example shows how to compute and retrieve ROC curves for a binary classification model.

import datarobot as dr
from datarobot.insights.roc_curve import RocCurve

model_id = model.id  # or model_id = 'YOUR_MODEL_ID'

# Compute a new ROC curve and wait for it to complete
roc = RocCurve.create(entity_id=model_id)  # default source is 'validation'

# Access ROC curve properties
print(roc.auc)
>>> 0.85
print(roc.kolmogorov_smirnov_metric)
>>> 0.42
print(roc.roc_points[:2])
>>> [{'accuracy': 0.539375, 'f1_score': 0.0, 'false_negative_score': 737, ...}, ...]

# Compute ROC curve on a different partition, and return immediately with job reference
job = RocCurve.compute(entity_id=model_id, source='holdout')
# Wait for the job to complete
roc_holdout = job.get_result_when_complete()
print(roc_holdout.auc)
>>> 0.83

# Get a pre-existing ROC curve (if already computed)
existing_roc = RocCurve.get(entity_id=model_id, source='validation')

# List all available ROC curves for a model
roc_list = RocCurve.list(entity_id=model_id)
print(roc_list)
>>> [<datarobot.insights.roc_curve.RocCurve object at 0x7fc0a7549f60>, ...]
print([(r.source, r.auc) for r in roc_list])
>>> [('validation', 0.85), ('holdout', 0.83)]

ROC Curve with data slices

You can compute ROC curves for specific data slices:

from datarobot.insights.roc_curve import RocCurve

# Get a pre-existing ROC curve for a specific data slice
roc_sliced = RocCurve.get(entity_id=model_id, source='validation', data_slice_id='slice_id_here')

# Or compute a new one
job = RocCurve.compute(entity_id=model_id, source='validation', data_slice_id='slice_id_here')
roc_sliced = job.get_result_when_complete()

Lift chart

Lift charts depict how well a model segments the target population and how well the model performs for different ranges of values of the target variable.

import datarobot as dr
from datarobot.insights.lift_chart import LiftChart

model_id = model.id  # or model_id = 'YOUR_MODEL_ID'

# Compute a new lift chart and wait for it to complete
lift = LiftChart.create(entity_id=model_id)  # default source is 'validation'

# Access lift chart bins
print(lift.bins[:3])
>>> [{'actual': 0.4, 'predicted': 0.227, 'bin_weight': 5.0}, ...]

# Compute lift chart on a different partition, and return immediately with job reference
job = LiftChart.compute(entity_id=model_id, source='holdout')
# Wait for the job to complete
lift_holdout = job.get_result_when_complete()

# Get a pre-existing lift chart (if already computed)
existing_lift = LiftChart.get(entity_id=model_id, source='validation')

# List all available lift charts for a model
lift_list = LiftChart.list(entity_id=model_id)
print(lift_list)
>>> [<datarobot.insights.lift_chart.LiftChart object at 0x7fe242eeaa10>, ...]
print([l.source for l in lift_list])
>>> ['validation', 'holdout', 'crossValidation']

# Get lift chart for a specific data slice
lift_sliced = LiftChart.get(entity_id=model_id, source='validation', data_slice_id='slice_id_here')

Residuals

The residuals charts show the distribution of prediction errors for regression models.

import datarobot as dr
from datarobot.insights.residuals import Residuals

model_id = model.id  # or model_id = 'YOUR_MODEL_ID'

# Compute a new residuals chart and wait for it to complete
residuals = Residuals.create(entity_id=model_id)  # default source is 'validation'

# Access residuals properties
print(residuals.coefficient_of_determination)
>>> 0.85
print(residuals.residual_mean)
>>> 0.023
print(residuals.standard_deviation)
>>> 1.42

# Access histogram data
print(residuals.histogram[:3])
>>> [{'interval_start': -33.37, 'interval_end': -32.52, 'occurrences': 1}, ...]

# Access raw chart data (actual, predicted, residual, row number)
print(residuals.chart_data[:3])
>>> [[45.2, 43.8, -1.4, 0], [52.1, 51.9, -0.2, 1], ...]

# Compute residuals on a different partition, and return immediately with job reference
job = Residuals.compute(entity_id=model_id, source='holdout')
# Wait for the job to complete
residuals_holdout = job.get_result_when_complete()

# Get a pre-existing residuals chart (if already computed)
existing_residuals = Residuals.get(entity_id=model_id, source='validation')

# List all available residuals charts for a model
residuals_list = Residuals.list(entity_id=model_id)
print(residuals_list)
>>> [<datarobot.insights.residuals.Residuals object at 0x7fbce8305ae0>, ...]
print([r.source for r in residuals_list])
>>> ['validation', 'holdout']

# Get residuals for a specific data slice
residuals_sliced = Residuals.get(entity_id=model_id, source='validation', data_slice_id='slice_id_here')