External scores and insights

class datarobot.ExternalScores

Metric scores on prediction dataset with target or actual value column in unsupervised case. Contains project metrics for supervised and special classification metrics set for unsupervised projects.

Added in version v2.21.

Variables:

  • project_id (str) – id of the project the model belongs to

  • model_id (str) – id of the model

  • dataset_id (str) – id of the prediction dataset with target or actual value column for unsupervised case

  • actual_value_column (Optional[str]) – For unsupervised projects only. Actual value column which was used to calculate the classification metrics and insights on the prediction dataset.

  • scores (list of dicts in a form of {'label': metric_name, 'value': score}) – Scores on the dataset.

Examples

List all scores for a dataset

from datarobot.models.external_dataset_scores_insights.external_scores import ExternalScores
scores = ExternalScores.list(project_id, dataset_id=dataset_id)
classmethod create(project_id, model_id, dataset_id, actual_value_column=None)

Compute an external dataset insights for the specified model.

Parameters:

  • project_id (str) – id of the project the model belongs to

  • model_id (str) – id of the model for which insights is requested

  • dataset_id (str) – id of the dataset for which insights is requested

  • actual_value_column (Optional[str]) – actual values column label, for unsupervised projects only

Returns:

job – an instance of created async job

Return type:

Job

classmethod list(project_id, model_id=None, dataset_id=None, offset=0, limit=100)

Fetch external scores list for the project and optionally for model and dataset.

Parameters:

  • project_id (str) – id of the project

  • model_id (Optional[str]) – if specified, only scores for this model will be retrieved

  • dataset_id (Optional[str]) – if specified, only scores for this dataset will be retrieved

  • offset (Optional[int]) – this many results will be skipped, default: 0

  • limit (Optional[int]) – at most this many results are returned, default: 100, max 1000. To return all results, specify 0

Return type:

List[ExternalScores]

Returns:

A list of External Scores objects

classmethod get(project_id, model_id, dataset_id)

Retrieve external scores for the project, model and dataset.

Parameters:

  • project_id (str) – id of the project

  • model_id (str) – if specified, only scores for this model will be retrieved

  • dataset_id (str) – if specified, only scores for this dataset will be retrieved

Return type:

ExternalScores

Returns:

External Scores object

class datarobot.ExternalLiftChart

Lift chart for the model and prediction dataset with target or actual value column in unsupervised case.

Added in version v2.21.

LiftChartBin is a dict containing the following:

  • actual (float) Sum of actual target values in bin

  • predicted (float) Sum of predicted target values in bin

  • bin_weight (float) The weight of the bin. For weighted projects, it is the sum of the weights of the rows in the bin. For unweighted projects, it is the number of rows in the bin.

Variables:

  • dataset_id (str) – id of the prediction dataset with target or actual value column for unsupervised case

  • bins (list of dict) – List of dicts with schema described as LiftChartBin above.

classmethod list(project_id, model_id, dataset_id=None, offset=0, limit=100)

Retrieve list of the lift charts for the model.

Parameters:

  • project_id (str) – id of the project

  • model_id (str) – if specified, only lift chart for this model will be retrieved

  • dataset_id (Optional[str]) – if specified, only lift chart for this dataset will be retrieved

  • offset (Optional[int]) – this many results will be skipped, default: 0

  • limit (Optional[int]) – at most this many results are returned, default: 100, max 1000. To return all results, specify 0

Return type:

List[ExternalLiftChart]

Returns:

A list of ExternalLiftChart objects

classmethod get(project_id, model_id, dataset_id)

Retrieve lift chart for the model and prediction dataset.

Parameters:

  • project_id (str) – project id

  • model_id (str) – model id

  • dataset_id (str) – prediction dataset id with target or actual value column for unsupervised case

Return type:

ExternalLiftChart

Returns:

ExternalLiftChart object

class datarobot.ExternalRocCurve

ROC curve data for the model and prediction dataset with target or actual value column in unsupervised case.

Added in version v2.21.

Variables:

  • dataset_id (str) – id of the prediction dataset with target or actual value column for unsupervised case

  • roc_points (list of dict) – List of precalculated metrics associated with thresholds for ROC curve.

  • negative_class_predictions (list of float) – List of predictions from example for negative class

  • positive_class_predictions (list of float) – List of predictions from example for positive class

classmethod list(project_id, model_id, dataset_id=None, offset=0, limit=100)

Retrieve list of the roc curves for the model.

Parameters:

  • project_id (str) – id of the project

  • model_id (str) – if specified, only lift chart for this model will be retrieved

  • dataset_id (Optional[str]) – if specified, only lift chart for this dataset will be retrieved

  • offset (Optional[int]) – this many results will be skipped, default: 0

  • limit (Optional[int]) – at most this many results are returned, default: 100, max 1000. To return all results, specify 0

Return type:

List[ExternalRocCurve]

Returns:

A list of ExternalRocCurve objects

classmethod get(project_id, model_id, dataset_id)

Retrieve ROC curve chart for the model and prediction dataset.

Parameters:

  • project_id (str) – project id

  • model_id (str) – model id

  • dataset_id (str) – prediction dataset id with target or actual value column for unsupervised case

Return type:

ExternalRocCurve

Returns:

ExternalRocCurve object