Custom metrics

class datarobot.models.deployment.custom_metrics.CustomMetric

A DataRobot custom metric.

Added in version v3.4.

Variables:

id (str) – The ID of the custom metric.
deployment_id (str) – The ID of the deployment.
name (str) – The name of the custom metric.
units (str) – The units, or the y-axis label, of the given custom metric.
baseline_values (BaselinesValues) – The baseline value used to add “reference dots” to the values over time chart.
is_model_specific (bool) – Determines whether the metric is related to the model or deployment.
type (CustomMetricAggregationType) – The aggregation type of the custom metric.
directionality (CustomMetricDirectionality) – The directionality of the custom metric.
time_step (CustomMetricBucketTimeStep) – Custom metric time bucket size.
description (str) – A description of the custom metric.
association_id (DatasetColumn) – A custom metric association_id column source when reading values from columnar dataset.
timestamp (DatasetColumn) – A custom metric timestamp column source when reading values from columnar dataset.
value (DatasetColumn) – A custom metric value source when reading values from columnar dataset.
sample_count (DatasetColumn) – A custom metric sample source when reading values from columnar dataset.
batch (str) – A custom metric batch ID source when reading values from columnar dataset.

classmethod create(name, deployment_id, units, is_model_specific, aggregation_type, time_step='hour', directionality=None, description=None, baseline_value=None, value_column_name=None, sample_count_column_name=None, timestamp_column_name=None, timestamp_format=None, batch_column_name=None, categories=None, is_geospatial=None, geospatial_segment_attribute=None)

Create a custom metric for a deployment

Parameters:

name (str) – The name of the custom metric.
deployment_id (str) – The id of the deployment.
units (str) – The units, or the y-axis label, of the given custom metric.
baseline_value (float) – The baseline value used to add “reference dots” to the values over time chart.
is_model_specific (bool) – Determines whether the metric is related to the model or deployment.
aggregation_type (CustomMetricAggregationType) – The aggregation type of the custom metric.
directionality (CustomMetricDirectionality) – The directionality of the custom metric.
time_step (CustomMetricBucketTimeStep) – Custom metric time bucket size.
description (Optional[str]) – A description of the custom metric.
value_column_name (Optional[str]) – A custom metric value column name when reading values from columnar dataset.
sample_count_column_name (Optional[str]) – Points to a weight column name if users provide pre-aggregated metric values from columnar dataset.
timestamp_column_name (Optional[str]) – A custom metric timestamp column name when reading values from columnar dataset.
timestamp_format (Optional[str]) – A custom metric timestamp format when reading values from columnar dataset.
batch_column_name (Optional[str]) – A custom metric batch ID column name when reading values from columnar dataset.
is_geospatial (Optional[bool]) – Determines whether the metric is geospatial or not.
geospatial_segment_attribute (Optional[str]) – The name of the geospatial segment attribute.

Returns:

The custom metric object.

Return type:

CustomMetric

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="Sample metric",
    units="Y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
    )

classmethod get(deployment_id, custom_metric_id)

Get a custom metric for a deployment

Parameters:

deployment_id (str) – The ID of the deployment.
custom_metric_id (str) – The ID of the custom metric.

Returns:

The custom metric object.

Return type:

CustomMetric

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

custom_metric.id
>>>'65f17bdcd2d66683cdfc1113'

classmethod list(deployment_id)

List all custom metrics for a deployment

Parameters:: deployment_id (str) – The ID of the deployment.
Returns:: custom_metrics – A list of custom metrics objects.
Return type:: list

Examples

from datarobot.models.deployment import CustomMetric

custom_metrics = CustomMetric.list(deployment_id="5c939e08962d741e34f609f0")
custom_metrics[0].id
>>>'65f17bdcd2d66683cdfc1113'

classmethod delete(deployment_id, custom_metric_id)

Delete a custom metric associated with a deployment.

Parameters:

deployment_id (str) – The ID of the deployment.
custom_metric_id (str) – The ID of the custom metric.

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

CustomMetric.delete(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

update(name=None, units=None, aggregation_type=None, directionality=None, time_step=None, description=None, baseline_value=None, value_column_name=None, sample_count_column_name=None, timestamp_column_name=None, timestamp_format=None, batch_column_name=None)

Update metadata of a custom metric

Parameters:

name (Optional[str]) – The name of the custom metric.
units (Optional[str]) – The units, or the y-axis label, of the given custom metric.
baseline_value (Optional[float]) – The baseline value used to add “reference dots” to the values over time chart.
aggregation_type (Optional[CustomMetricAggregationType]) – The aggregation type of the custom metric.
directionality (Optional[CustomMetricDirectionality]) – The directionality of the custom metric.
time_step (Optional[CustomMetricBucketTimeStep]) – Custom metric time bucket size.
description (Optional[str]) – A description of the custom metric.
value_column_name (Optional[str]) – A custom metric value column name when reading values from columnar dataset.
sample_count_column_name (Optional[str]) – Points to a weight column name if users provide pre-aggregated metric values from columnar dataset.
timestamp_column_name (Optional[str]) – A custom metric timestamp column name when reading values from columnar dataset.
timestamp_format (Optional[str]) – A custom metric timestamp format when reading values from columnar dataset.
batch_column_name (Optional[str]) – A custom metric batch ID column name when reading values from columnar dataset.

Returns:

The custom metric object.

Return type:

CustomMetric

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
custom_metric = custom_metric.update(
    deployment_id="5c939e08962d741e34f609f0",
    name="Sample metric",
    units="Y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
    )

unset_baseline()

Unset the baseline value of a custom metric

Return type:: None

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
custom_metric.baseline_values
>>> [{'value': 12.0}]
custom_metric.unset_baseline()
custom_metric.baseline_values
>>> []

submit_values(data, model_id=None, model_package_id=None, dry_run=False)

Submit aggregated custom metrics values from JSON.

Parameters:

data (pd.DataFrame or List[CustomMetricBucket]) – The data containing aggregated custom metric values.
model_id (Optional[str]) – For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.
model_package_id (Optional[str]) – For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.
dry_run (Optional[bool]) – Specifies whether or not metric data is submitted in production mode (where data is saved).

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# data for values over time
data = [{
    'value': 12,
    'sample_size': 3,
    'timestamp': '2024-03-15T14:00:00'
}]

# data witch association ID
data = [{
    'value': 12,
    'sample_size': 3,
    'timestamp': '2024-03-15T14:00:00',
    'association_id': '65f44d04dbe192b552e752ed'
}]

# data for batches
data = [{
    'value': 12,
    'sample_size': 3,
    'batch': '65f44c93fedc5de16b673a0d'
}]

# for deployment specific metrics
custom_metric.submit_values(data=data)

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25")

# dry run
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25", dry_run=True)

submit_single_value(value, model_id=None, model_package_id=None, dry_run=False, segments=None)

Submit a single custom metric value at the current moment.

Parameters:

value (float) – Single numeric custom metric value.
model_id (Optional[str]) – For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.
model_package_id (Optional[str]) – For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.
dry_run (Optional[bool]) – Specifies whether or not metric data is submitted in production mode (where data is saved).
segments (Optional[CustomMetricSegmentFromJSON]) – A list of segments for a custom metric used in segmented analysis.

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# for deployment specific metrics
custom_metric.submit_single_value(value=121)

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25")

# dry run
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", dry_run=True)

# for segmented analysis
segments = [{"name": "custom_seg", "value": "val_1"}]
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", segments=segments)

submit_values_from_catalog(dataset_id, model_id=None, model_package_id=None, batch_id=None, segments=None, geospatial=None)

Submit aggregated custom metrics values from dataset (AI catalog). The names of the columns in the dataset should correspond to the names of the columns that were defined in the custom metric. In addition, the format of the timestamps should also be the same as defined in the metric.

Parameters:

dataset_id (str) – The ID of the source dataset.
model_id (Optional[str]) – For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.
model_package_id (Optional[str]) – For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.
batch_id (Optional[str]) – Specifies a batch ID associated with all values provided by this dataset, an alternative to providing batch IDs as a column within a dataset (at the record level).
segments (Optional[CustomMetricSegmentFromDataset]) – A list of segments for a custom metric used in segmented analysis.
geospatial (Optional[Geospatial]) – A geospatial column source when reading values from columnar dataset.

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# for deployment specific metrics
custom_metric.submit_values_from_catalog(dataset_id="61093144cabd630828bca321")

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25"
)

# for segmented analysis
segments = [{"name": "custom_seg", "column": "column_with_segment_values"}]
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25",
    segments=segments
)

get_values_over_time(start, end, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None, bucket_size='P7D')

Retrieve values of a single custom metric over a time period.

Parameters:

start (datetime or str) – Start of the time period.
end (datetime or str) – End of the time period.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
bucket_size (Optional[str]) – Time duration of a bucket, in ISO 8601 time duration format.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_over_time – The queried custom metric values over time information.

Return type:

CustomMetricValuesOverTime

Examples

from datarobot.models.deployment import CustomMetric
from datetime import datetime, timedelta

now=datetime.now()
custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
values_over_time = custom_metric.get_values_over_time(start=now - timedelta(days=7), end=now)

values_over_time.bucket_values
>>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1.0,
>>> datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 123.0}}

values_over_time.bucket_sample_sizes
>>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1,
>>>  datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 1}}

values_over_time.get_buckets_as_dataframe()
>>>                        start                       end  value  sample_size
>>> 0  2024-03-21 16:00:00+00:00 2024-03-21 17:00:00+00:00    NaN          NaN
>>> 1  2024-03-21 17:00:00+00:00 2024-03-21 18:00:00+00:00    NaN          NaN

get_values_over_space(start=None, end=None, model_id=None, model_package_id=None)

Retrieve values of a custom metric over space.

Parameters:

start (Optional[datetime]) – Start of the time period.
end (Optional[datetime]) – End of the time period.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.

Returns:

custom_metric_over_space – The queried custom metric values over space information.

Return type:

CustomMetricValuesOverSpace

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
values_over_space = custom_metric.get_values_over_space(model_package_id='6421df32525c58cc6f991f25')

get_summary(start, end, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a time period.

Parameters:

start (datetime or str) – Start of the time period.
end (datetime or str) – End of the time period.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_summary – The summary of the custom metric.

Return type:

CustomMetricSummary

Examples

from datarobot.models.deployment import CustomMetric
from datetime import datetime, timedelta

now=datetime.now()
custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
summary = custom_metric.get_summary(start=now - timedelta(days=7), end=now)

print(summary)
>> "CustomMetricSummary(2024-03-21 15:52:13.392178+00:00 - 2024-03-22 15:52:13.392168+00:00:
{'id': '65fd9b1c0c1a840bc6751ce0', 'name': 'Test METRIC', 'value': 215.0, 'sample_count': 13,
'baseline_value': 12.0, 'percent_change': 24.02})"

get_values_over_batch(batch_ids=None, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve values of a single custom metric over batches.

Parameters:

batch_ids (Optional[List[str]]) – Specify a list of batch IDs to pull the data for.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_over_batch – The queried custom metric values over batch information.

Return type:

CustomMetricValuesOverBatch

Examples

   from datarobot.models.deployment import CustomMetric

   custom_metric = CustomMetric.get(
       deployment_id="5c939e08962d741e34f609f0",
       custom_metric_id="65f17bdcd2d66683cdfc1113"
   )
   # all batch metrics all model specific
   values_over_batch = custom_metric.get_values_over_batch(model_package_id='6421df32525c58cc6f991f25')

   values_over_batch.bucket_values
   >>> {'6572db2c9f9d4ad3b9de33d0': 35.0, '6572db2c9f9d4ad3b9de44e1': 105.0}

   values_over_batch.bucket_sample_sizes
   >>> {'6572db2c9f9d4ad3b9de33d0': 6, '6572db2c9f9d4ad3b9de44e1': 8}

values_over_batch.get_buckets_as_dataframe()
>>>                    batch_id                     batch_name  value  sample_size
>>> 0  6572db2c9f9d4ad3b9de33d0  Batch 1 - 03/26/2024 13:04:46   35.0            6
>>> 1  6572db2c9f9d4ad3b9de44e1  Batch 2 - 03/26/2024 13:06:04  105.0            8

get_batch_summary(batch_ids=None, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a batch.

Parameters:

batch_ids (Optional[List[str]]) – Specify a list of batch IDs to pull the data for.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_summary – The batch summary of the custom metric.

Return type:

CustomMetricBatchSummary

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
# all batch metrics all model specific
batch_summary = custom_metric.get_batch_summary(model_package_id='6421df32525c58cc6f991f25')

print(batch_summary)
>> CustomMetricBatchSummary({'id': '6605396413434b3a7b74342c', 'name': 'batch metric', 'value': 41.25,
'sample_count': 28, 'baseline_value': 123.0, 'percent_change': -66.46})

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverTime

Custom metric over time information.

Added in version v3.4.

Variables:

buckets (List[Bucket]) – A list of bucketed time periods and the custom metric values aggregated over that period.
summary (Summary) – The summary of values over time retrieval.
metric (Dict) – A custom metric definition.
deployment_id (str) – The ID of the deployment.
segment_attribute (str) – The name of the segment on which segment analysis is being performed.
segment_value (str) – The value of the segment_attribute to segment on.

classmethod get(deployment_id, custom_metric_id, start, end, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None, bucket_size='P7D')

Retrieve values of a single custom metric over a time period.

Parameters:

custom_metric_id (str) – The ID of the custom metric.
deployment_id (str) – The ID of the deployment.
start (datetime or str) – Start of the time period.
end (datetime or str) – End of the time period.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
bucket_size (Optional[str]) – Time duration of a bucket, in ISO 8601 time duration format.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_over_time – The queried custom metric values over time information.

Return type:

CustomMetricValuesOverTime

property bucket_values: Dict[datetime, int]

The metric value for all time buckets, keyed by start time of the bucket.

Returns:: bucket_values
Return type:: Dict

property bucket_sample_sizes: Dict[datetime, int]

The sample size for all time buckets, keyed by start time of the bucket.

Returns:: bucket_sample_sizes
Return type:: Dict

get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

Returns:: buckets
Return type:: pd.DataFrame

class datarobot.models.deployment.custom_metrics.CustomMetricSummary

The summary of a custom metric.

Added in version v3.4.

Variables:

period (Period) – A time period defined by a start and end tie
metric (Dict) – The summary of the custom metric.

classmethod get(deployment_id, custom_metric_id, start, end, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a time period.

Parameters:

custom_metric_id (str) – The ID of the custom metric.
deployment_id (str) – The ID of the deployment.
start (datetime or str) – Start of the time period.
end (datetime or str) – End of the time period.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_summary – The summary of the custom metric.

Return type:

CustomMetricSummary

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverBatch

Custom metric over batch information.

Added in version v3.4.

Variables:

buckets (List[BatchBucket]) – A list of buckets with custom metric values aggregated over batches.
metric (Dict) – A custom metric definition.
deployment_id (str) – The ID of the deployment.
segment_attribute (str) – The name of the segment on which segment analysis is being performed.
segment_value (str) – The value of the segment_attribute to segment on.

classmethod get(deployment_id, custom_metric_id, batch_ids=None, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve values of a single custom metric over batches.

Parameters:

custom_metric_id (str) – The ID of the custom metric.
deployment_id (str) – The ID of the deployment.
batch_ids (Optional[List[str]]) – Specify a list of batch IDs to pull the data for.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_over_batch – The queried custom metric values over batch information.

Return type:

CustomMetricValuesOverBatch

property bucket_values: Dict[str, int]

The metric value for all batch buckets, keyed by batch ID

Returns:: bucket_values
Return type:: Dict

property bucket_sample_sizes: Dict[str, int]

The sample size for all batch buckets, keyed by batch ID.

Returns:: bucket_sample_sizes
Return type:: Dict

get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

Returns:: buckets
Return type:: pd.DataFrame

class datarobot.models.deployment.custom_metrics.CustomMetricBatchSummary

The batch summary of a custom metric.

Added in version v3.4.

Variables:: metric (Dict) – The summary of the batch custom metric.

classmethod get(deployment_id, custom_metric_id, batch_ids=None, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a batch.

Parameters:

custom_metric_id (str) – The ID of the custom metric.
deployment_id (str) – The ID of the deployment.
batch_ids (Optional[List[str]]) – Specify a list of batch IDs to pull the data for.
model_id (Optional[str]) – The ID of the model.
model_package_id (Optional[str]) – The ID of the model package.
segment_attribute (Optional[str]) – The name of the segment on which segment analysis is being performed.
segment_value (Optional[str]) – The value of the segment_attribute to segment on.

Returns:

custom_metric_summary – The batch summary of the custom metric.

Return type:

CustomMetricBatchSummary

class datarobot.models.deployment.custom_metrics.HostedCustomMetricTemplate

Template for hosted custom metric.

classmethod list(search=None, order_by=None, metric_type=None, offset=None, limit=None)

List all hosted custom metric templates.

Parameters:

search (Optional[str]) – Search string.
order_by (Optional[ListHostedCustomMetricTemplatesSortQueryParams]) – Ordering field.
metric_type (Optional[HostedCustomMetricsTemplateMetricTypeQueryParams]) – Type of the metric.
offset (Optional[int]) – Offset for pagination.
limit (Optional[int]) – Limit for pagination.

Returns:

templates

Return type:

List[HostedCustomMetricTemplate]

classmethod get(template_id)

Get a hosted custom metric template by ID.

Parameters:: template_id (str) – ID of the template.
Returns:: template
Return type:: HostedCustomMetricTemplate

class datarobot.models.deployment.custom_metrics.HostedCustomMetric

Hosted custom metric.

classmethod list(job_id, skip=None, limit=None)

List all hosted custom metrics for a job.

Parameters:: job_id (str) – ID of the job.
Returns:: metrics
Return type:: List[HostedCustomMetric]

classmethod create_from_template(template_id, deployment_id, job_name, custom_metric_name, job_description=None, custom_metric_description=None, sidecar_deployment_id=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Create a hosted custom metric from a template. A shortcut for 2 calls: Job.from_custom_metric_template(template_id) HostedCustomMetrics.create_from_custom_job()

Parameters:

template_id (str) – ID of the template.
deployment_id (str) – ID of the deployment.
job_name (str) – Name of the job.
custom_metric_name (str) – Name of the metric.
job_description (Optional[str]) – Description of the job.
custom_metric_description (Optional[str]) – Description of the metric.
sidecar_deployment_id (Optional[str]) – ID of the sidecar deployment.
baseline_value (Optional[float]) – Baseline value.
timestamp (Optional[MetricTimestampSpoofing]) – Timestamp details.
value (Optional[ValueField]) – Value details.
sample_count (Optional[SampleCountField]) – Sample count details.
batch (Optional[BatchField]) – Batch details.
schedule (Optional[Schedule]) – Schedule details.
parameter_overrides (Optional[List[RuntimeParameterValue]]) – Parameter overrides.

Returns:

metric

Return type:

HostedCustomMetric

classmethod create_from_custom_job(custom_job_id, deployment_id, name, description=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None, geospatial_segment_attribute=None)

Create a hosted custom metric from existing custom job.

Parameters:

custom_job_id (str) – ID of the custom job.
deployment_id (str) – ID of the deployment.
name (str) – Name of the metric.
description (Optional[str]) – Description of the metric.
baseline_value (Optional[float]) – Baseline value.
timestamp (Optional[MetricTimestampSpoofing]) – Timestamp details.
value (Optional[ValueField]) – Value details.
sample_count (Optional[SampleCountField]) – Sample count details.
batch (Optional[BatchField]) – Batch details.
schedule (Optional[Schedule]) – Schedule details.
parameter_overrides (Optional[List[RuntimeParameterValue]]) – Parameter overrides.
geospatial_segment_attribute (Optional[str]) – The name of the geospatial segment attribute. Only applicable for geospatial custom metrics.

Returns:

metric

Return type:

HostedCustomMetric

update(name=None, description=None, units=None, directionality=None, aggregation_type=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Update the hosted custom metric.

Parameters:

name (Optional[str]) – Name of the metric.
description (Optional[str]) – Description of the metric.
units (Optional[str]) – Units of the metric.
directionality (Optional[str]) – Directionality of the metric.
aggregation_type (Optional[CustomMetricAggregationType]) – Aggregation type of the metric.
baseline_value (Optional[float]) – Baseline values.
timestamp (Optional[MetricTimestampSpoofing]) – Timestamp details.
value (Optional[ValueField]) – Value details.
sample_count (Optional[SampleCountField]) – Sample count details.
batch (Optional[BatchField]) – Batch details.
schedule (Optional[Schedule]) – Schedule details.
parameter_overrides (Optional[List[RuntimeParameterValue]]) – Parameter overrides.

Returns:

metric

Return type:

HostedCustomMetric

delete()

Delete the hosted custom metric.

Return type:: None

class datarobot.models.deployment.custom_metrics.DeploymentDetails: Information about a hosted custom metric deployment.

class datarobot.models.deployment.custom_metrics.MetricBaselineValue: The baseline values for a custom metric.

class datarobot.models.deployment.custom_metrics.SampleCountField: A weight column used with columnar datasets if pre-aggregated metric values are provided.

class datarobot.models.deployment.custom_metrics.ValueField: A custom metric value source for when reading values from a columnar dataset like a file.

class datarobot.models.deployment.custom_metrics.MetricTimestampSpoofing: Custom metric timestamp spoofing. Occurs when reading values from a file, like a dataset. By default, replicates pd.to_datetime formatting behavior.

class datarobot.models.deployment.custom_metrics.BatchField: A custom metric batch ID source for when reading values from a columnar dataset like a file.

class datarobot.models.deployment.custom_metrics.HostedCustomMetricBlueprint

Hosted custom metric blueprints provide an option to share custom metric settings between multiple custom metrics sharing the same custom jobs. When a custom job of a hosted custom metric type is connected to the deployment, all the custom metric parameters from the blueprint are automatically copied.

classmethod get(custom_job_id)

Get a hosted custom metric blueprint.

Parameters:: custom_job_id (str) – ID of the custom job.
Returns:: blueprint
Return type:: HostedCustomMetricBlueprint

classmethod create(custom_job_id, directionality, units, type, time_step, is_model_specific, is_geospatial=None)

Create a hosted custom metric blueprint.

Parameters:

custom_job_id (str) – ID of the custom job.
directionality (str) – Directionality of the metric.
units (str) – Units of the metric.
type (str) – Type of the metric.
time_step (str) – Time step of the metric.
is_model_specific (bool) – Whether the metric is model specific.
is_geospatial (Optional[bool]) – Determines whether the metric is geospatial.

Returns:

blueprint

Return type:

HostedCustomMetricBlueprint

update(directionality=None, units=None, type=None, time_step=None, is_model_specific=None, is_geospatial=None)

Update a hosted custom metric blueprint.

Parameters:

directionality (Optional[str]) – Directionality of the metric.
units (Optional[str]) – Units of the metric.
type (Optional[str]) – Type of the metric.
time_step (Optional[str]) – Time step of the metric.
is_model_specific (Optional[bool]) – Determines whether the metric is model specific.
is_geospatial (Optional[bool]) – Determines whether the metric is geospatial.

Returns:

updated_blueprint

Return type:

HostedCustomMetricBlueprint

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverSpace

Custom metric values over space.

Added in version v3.7.

Variables:

buckets (List[BatchBucket]) – A list of buckets with custom metric values aggregated over geospatial hexagons.
metric (Dict) – A custom metric definition.
model_id (str) – The ID of the model.
model_package_id (str) – The ID of the model package (also known as registered model version id).
summary (Dict) – Start-end interval over which data is retrieved.

classmethod get(deployment_id, custom_metric_id, start=None, end=None, model_package_id=None, model_id=None)

Retrieve custom metric values over space.

Parameters:

deployment_id (str) – The id of the deployment.
custom_metric_id (str) – The id of the custom metric.
start (datetime) – The start time of the interval.
end (datetime) – The end time of the interval.
model_package_id (str) – The id of the model package.
model_id (str) – The id of the model.

Returns:

values_over_space – Custom metric values over geospatial hexagons.

Return type:

CustomMetricValuesOverSpace