Custom Metrics

class datarobot.models.deployment.custom_metrics.CustomMetric(id, name, units, is_model_specific, type, time_step='hour', directionality=None, baseline_values=None, description=None, association_id=None, value=None, sample_count=None, timestamp=None, batch=None, deployment_id=None, categories=None)

A DataRobot custom metric.

Added in version v3.4.

Attributes:

id: str

The ID of the custom metric.

deployment_id: str

The ID of the deployment.

name: str

The name of the custom metric.

units: str

The units, or the y-axis label, of the given custom metric.

baseline_values: BaselinesValues

The baseline value used to add “reference dots” to the values over time chart.

is_model_specific: bool

Determines whether the metric is related to the model or deployment.

type: CustomMetricAggregationType

The aggregation type of the custom metric.

directionality: CustomMetricDirectionality

The directionality of the custom metric.

time_step: CustomMetricBucketTimeStep

Custom metric time bucket size.

description: str

A description of the custom metric.

association_id: DatasetColumn

A custom metric association_id column source when reading values from columnar dataset.

timestamp: DatasetColumn

A custom metric timestamp column source when reading values from columnar dataset.

value: DatasetColumn

A custom metric value source when reading values from columnar dataset.

sample_count: DatasetColumn

A custom metric sample source when reading values from columnar dataset.

batch: str

A custom metric batch ID source when reading values from columnar dataset.

classmethod create(name, deployment_id, units, is_model_specific, aggregation_type, time_step='hour', directionality=None, description=None, baseline_value=None, value_column_name=None, sample_count_column_name=None, timestamp_column_name=None, timestamp_format=None, batch_column_name=None, categories=None)

Create a custom metric for a deployment

Parameters:

name: str

The name of the custom metric.

deployment_id: str

The id of the deployment.

units: str

The units, or the y-axis label, of the given custom metric.

baseline_value: float

The baseline value used to add “reference dots” to the values over time chart.

is_model_specific: bool

Determines whether the metric is related to the model or deployment.

aggregation_type: CustomMetricAggregationType

The aggregation type of the custom metric.

directionality: CustomMetricDirectionality

The directionality of the custom metric.

time_step: CustomMetricBucketTimeStep

Custom metric time bucket size.

description: Optional[str]

A description of the custom metric.

value_column_name: Optional[str]

A custom metric value column name when reading values from columnar dataset.

sample_count_column_name: Optional[str]

Points to a weight column name if users provide pre-aggregated metric values from columnar dataset.

timestamp_column_name: Optional[str]

A custom metric timestamp column name when reading values from columnar dataset.

timestamp_format: Optional[str]

A custom metric timestamp format when reading values from columnar dataset.

batch_column_name: Optional[str]

A custom metric batch ID column name when reading values from columnar dataset.

Returns:

CustomMetric

The custom metric object.

Return type:

CustomMetric

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="Sample metric",
    units="Y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
    )

classmethod get(deployment_id, custom_metric_id)

Get a custom metric for a deployment

Parameters:

deployment_id: str

The ID of the deployment.

custom_metric_id: str

The ID of the custom metric.

Returns:

CustomMetric

The custom metric object.

Return type:

CustomMetric

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

custom_metric.id
>>>'65f17bdcd2d66683cdfc1113'

classmethod list(deployment_id)

List all custom metrics for a deployment

Parameters:

deployment_id: str

The ID of the deployment.

Returns:

custom_metrics: list

A list of custom metrics objects.

Return type:

List[CustomMetric]

Examples

from datarobot.models.deployment import CustomMetric

custom_metrics = CustomMetric.list(deployment_id="5c939e08962d741e34f609f0")
custom_metrics[0].id
>>>'65f17bdcd2d66683cdfc1113'

classmethod delete(deployment_id, custom_metric_id)

Delete a custom metric associated with a deployment.

Parameters:

deployment_id: str

The ID of the deployment.

custom_metric_id: str

The ID of the custom metric.

Returns:

None

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

CustomMetric.delete(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

update(name=None, units=None, aggregation_type=None, directionality=None, time_step=None, description=None, baseline_value=None, value_column_name=None, sample_count_column_name=None, timestamp_column_name=None, timestamp_format=None, batch_column_name=None)

Update metadata of a custom metric

Parameters:

name: Optional[str]

The name of the custom metric.

units: Optional[str]

The units, or the y-axis label, of the given custom metric.

baseline_value: Optional[float]

The baseline value used to add “reference dots” to the values over time chart.

aggregation_type: Optional[CustomMetricAggregationType]

The aggregation type of the custom metric.

directionality: Optional[CustomMetricDirectionality]

The directionality of the custom metric.

time_step: Optional[CustomMetricBucketTimeStep]

Custom metric time bucket size.

description: Optional[str]

A description of the custom metric.

value_column_name: Optional[str]

A custom metric value column name when reading values from columnar dataset.

sample_count_column_name: Optional[str]

Points to a weight column name if users provide pre-aggregated metric values from columnar dataset.

timestamp_column_name: Optional[str]

A custom metric timestamp column name when reading values from columnar dataset.

timestamp_format: Optional[str]

A custom metric timestamp format when reading values from columnar dataset.

batch_column_name: Optional[str]

A custom metric batch ID column name when reading values from columnar dataset.

Returns:

CustomMetric

The custom metric object.

Return type:

CustomMetric

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
custom_metric = custom_metric.update(
    deployment_id="5c939e08962d741e34f609f0",
    name="Sample metric",
    units="Y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER
    )

unset_baseline()

Unset the baseline value of a custom metric

Returns:

None

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
custom_metric.baseline_values
>>> [{'value': 12.0}]
custom_metric.unset_baseline()
custom_metric.baseline_values
>>> []

submit_values(data, model_id=None, model_package_id=None, dry_run=False, segments=None)

Submit aggregated custom metrics values from JSON.

Parameters:

data: pd.DataFrame or List[CustomMetricBucket]

The data containing aggregated custom metric values.

model_id: Optional[str]

For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.

model_package_id: Optional[str]

For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.

dry_run: Optional[bool]

Specifies whether or not metric data is submitted in production mode (where data is saved).

segments: Optional[CustomMetricSegmentFromJSON]

A list of segments for a custom metric used in segmented analysis.

Returns:

None

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# data for values over time
data = [{
    'value': 12,
    'sample_size': 3,
    'timestamp': '2024-03-15T14:00:00'
}]

# data witch association ID
data = [{
    'value': 12,
    'sample_size': 3,
    'timestamp': '2024-03-15T14:00:00',
    'association_id': '65f44d04dbe192b552e752ed'
}]

# data for batches
data = [{
    'value': 12,
    'sample_size': 3,
    'batch': '65f44c93fedc5de16b673a0d'
}]

# for deployment specific metrics
custom_metric.submit_values(data=data)

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25")

# dry run
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25", dry_run=True)

# for segmented analysis
segments = [{"name": "custom_seg", "value": "val_1"}]
custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25", segments=segments)

submit_single_value(value, model_id=None, model_package_id=None, dry_run=False, segments=None)

Submit a single custom metric value at the current moment.

Parameters:

value: float

Single numeric custom metric value.

model_id: Optional[str]

For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.

model_package_id: Optional[str]

For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.

dry_run: Optional[bool]

Specifies whether or not metric data is submitted in production mode (where data is saved).

segments: Optional[CustomMetricSegmentFromJSON]

A list of segments for a custom metric used in segmented analysis.

Returns:

None

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# for deployment specific metrics
custom_metric.submit_single_value(value=121)

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25")

# dry run
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", dry_run=True)

# for segmented analysis
segments = [{"name": "custom_seg", "value": "val_1"}]
custom_metric.submit_single_value(value=121, model_package_id="6421df32525c58cc6f991f25", segments=segments)

submit_values_from_catalog(dataset_id, model_id=None, model_package_id=None, batch_id=None, segments=None)

Submit aggregated custom metrics values from dataset (AI catalog). The names of the columns in the dataset should correspond to the names of the columns that were defined in the custom metric. In addition, the format of the timestamps should also be the same as defined in the metric.

Parameters:

dataset_id: str

The ID of the source dataset.

model_id: Optional[str]

For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model is not needed.

model_package_id: Optional[str]

For a model metric: the ID of the associated champion/challenger model, used to update the metric values. For a deployment metric: the ID of the model package is not needed.

batch_id: Optional[str]

Specifies a batch ID associated with all values provided by this dataset, an alternative to providing batch IDs as a column within a dataset (at the record level).

segments: Optional[CustomMetricSegmentFromDataset]

A list of segments for a custom metric used in segmented analysis.

Returns:

None

Return type:

None

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)

# for deployment specific metrics
custom_metric.submit_values_from_catalog(dataset_id="61093144cabd630828bca321")

# for model specific metrics pass model_package_id or model_id
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25"
)

# for segmented analysis
segments = [{"name": "custom_seg", "column": "column_with_segment_values"}]
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25",
    segments=segments
)

get_values_over_time(start, end, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None, bucket_size='P7D')

Retrieve values of a single custom metric over a time period.

Parameters:

start: datetime or str

Start of the time period.

end: datetime or str

End of the time period.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

bucket_size: Optional[str]

Time duration of a bucket, in ISO 8601 time duration format.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_over_time: CustomMetricValuesOverTime

The queried custom metric values over time information.

Return type:

CustomMetricValuesOverTime

Examples

from datarobot.models.deployment import CustomMetric
from datetime import datetime, timedelta

now=datetime.now()
custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
values_over_time = custom_metric.get_values_over_time(start=now - timedelta(days=7), end=now)

values_over_time.bucket_values
>>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1.0,
>>> datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 123.0}}

values_over_time.bucket_sample_sizes
>>> {datetime.datetime(2024, 3, 22, 14, 0, tzinfo=tzutc()): 1,
>>>  datetime.datetime(2024, 3, 22, 15, 0, tzinfo=tzutc()): 1}}

values_over_time.get_buckets_as_dataframe()
>>>                        start                       end  value  sample_size
>>> 0  2024-03-21 16:00:00+00:00 2024-03-21 17:00:00+00:00    NaN          NaN
>>> 1  2024-03-21 17:00:00+00:00 2024-03-21 18:00:00+00:00    NaN          NaN

get_summary(start, end, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a time period.

Parameters:

start: datetime or str

Start of the time period.

end: datetime or str

End of the time period.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_summary: CustomMetricSummary

The summary of the custom metric.

Return type:

CustomMetricSummary

Examples

from datarobot.models.deployment import CustomMetric
from datetime import datetime, timedelta

now=datetime.now()
custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
summary = custom_metric.get_summary(start=now - timedelta(days=7), end=now)

print(summary)
>> "CustomMetricSummary(2024-03-21 15:52:13.392178+00:00 - 2024-03-22 15:52:13.392168+00:00:
{'id': '65fd9b1c0c1a840bc6751ce0', 'name': 'Test METRIC', 'value': 215.0, 'sample_count': 13,
'baseline_value': 12.0, 'percent_change': 24.02})"

get_values_over_batch(batch_ids=None, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve values of a single custom metric over batches.

Parameters:

batch_idsOptional[List[str]]

Specify a list of batch IDs to pull the data for.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_over_batch: CustomMetricValuesOverBatch

The queried custom metric values over batch information.

Return type:

CustomMetricValuesOverBatch

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
# all batch metrics all model specific
values_over_batch = custom_metric.get_values_over_batch(model_package_id='6421df32525c58cc6f991f25')

values_over_batch.bucket_values
>>> {'6572db2c9f9d4ad3b9de33d0': 35.0, '6572db2c9f9d4ad3b9de44e1': 105.0}

values_over_batch.bucket_sample_sizes
>>> {'6572db2c9f9d4ad3b9de33d0': 6, '6572db2c9f9d4ad3b9de44e1': 8}

values_over_batch.get_buckets_as_dataframe()
>>>                    batch_id                     batch_name  value  sample_size
>>> 0  6572db2c9f9d4ad3b9de33d0  Batch 1 - 03/26/2024 13:04:46   35.0            6
>>> 1  6572db2c9f9d4ad3b9de44e1  Batch 2 - 03/26/2024 13:06:04  105.0            8

get_batch_summary(batch_ids=None, model_package_id=None, model_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a batch.

Parameters:

batch_idsOptional[List[str]]

Specify a list of batch IDs to pull the data for.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_summary: CustomMetricBatchSummary

The batch summary of the custom metric.

Return type:

CustomMetricBatchSummary

Examples

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
# all batch metrics all model specific
batch_summary = custom_metric.get_batch_summary(model_package_id='6421df32525c58cc6f991f25')

print(batch_summary)
>> CustomMetricBatchSummary({'id': '6605396413434b3a7b74342c', 'name': 'batch metric', 'value': 41.25,
'sample_count': 28, 'baseline_value': 123.0, 'percent_change': -66.46})

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverTime(buckets=None, summary=None, metric=None, deployment_id=None, segment_attribute=None, segment_value=None)

Custom metric over time information.

Added in version v3.4.

Attributes:

buckets: List[Bucket]

A list of bucketed time periods and the custom metric values aggregated over that period.

summary: Summary

The summary of values over time retrieval.

metric: Dict

A custom metric definition.

deployment_id: str

The ID of the deployment.

segment_attribute: str

The name of the segment on which segment analysis is being performed.

segment_value: str

The value of the segment_attribute to segment on.

classmethod get(deployment_id, custom_metric_id, start, end, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None, bucket_size='P7D')

Retrieve values of a single custom metric over a time period.

Parameters:

custom_metric_id: str

The ID of the custom metric.

deployment_id: str

The ID of the deployment.

start: datetime or str

Start of the time period.

end: datetime or str

End of the time period.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

bucket_size: Optional[str]

Time duration of a bucket, in ISO 8601 time duration format.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_over_time: CustomMetricValuesOverTime

The queried custom metric values over time information.

Return type:

CustomMetricValuesOverTime

property bucket_values: Dict[datetime, int]

The metric value for all time buckets, keyed by start time of the bucket.

Returns:

bucket_values: Dict

property bucket_sample_sizes: Dict[datetime, int]

The sample size for all time buckets, keyed by start time of the bucket.

Returns:

bucket_sample_sizes: Dict

get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

Returns:

buckets: pd.DataFrame

Return type:

DataFrame

class datarobot.models.deployment.custom_metrics.CustomMetricSummary(period, metric, deployment_id=None)

The summary of a custom metric.

Added in version v3.4.

Attributes:

period: Period

A time period defined by a start and end tie

metric: Dict

The summary of the custom metric.

classmethod get(deployment_id, custom_metric_id, start, end, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a time period.

Parameters:

custom_metric_id: str

The ID of the custom metric.

deployment_id: str

The ID of the deployment.

start: datetime or str

Start of the time period.

end: datetime or str

End of the time period.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_summary: CustomMetricSummary

The summary of the custom metric.

Return type:

CustomMetricSummary

class datarobot.models.deployment.custom_metrics.CustomMetricValuesOverBatch(buckets=None, metric=None, deployment_id=None, segment_attribute=None, segment_value=None)

Custom metric over batch information.

Added in version v3.4.

Attributes:

buckets: List[BatchBucket]

A list of buckets with custom metric values aggregated over batches.

metric: Dict

A custom metric definition.

deployment_id: str

The ID of the deployment.

segment_attribute: str

The name of the segment on which segment analysis is being performed.

segment_value: str

The value of the segment_attribute to segment on.

classmethod get(deployment_id, custom_metric_id, batch_ids=None, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve values of a single custom metric over batches.

Parameters:

custom_metric_id: str

The ID of the custom metric.

deployment_id: str

The ID of the deployment.

batch_idsOptional[List[str]]

Specify a list of batch IDs to pull the data for.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_over_batch: CustomMetricValuesOverBatch

The queried custom metric values over batch information.

Return type:

CustomMetricValuesOverBatch

property bucket_values: Dict[str, int]

The metric value for all batch buckets, keyed by batch ID

Returns:

bucket_values: Dict

property bucket_sample_sizes: Dict[str, int]

The sample size for all batch buckets, keyed by batch ID.

Returns:

bucket_sample_sizes: Dict

get_buckets_as_dataframe()

Retrieves all custom metrics buckets in a pandas DataFrame.

Returns:

buckets: pd.DataFrame

Return type:

DataFrame

class datarobot.models.deployment.custom_metrics.CustomMetricBatchSummary(metric, deployment_id=None)

The batch summary of a custom metric.

Added in version v3.4.

Attributes:

metric: Dict

The summary of the batch custom metric.

classmethod get(deployment_id, custom_metric_id, batch_ids=None, model_id=None, model_package_id=None, segment_attribute=None, segment_value=None)

Retrieve the summary of a custom metric over a batch.

Parameters:

custom_metric_id: str

The ID of the custom metric.

deployment_id: str

The ID of the deployment.

batch_idsOptional[List[str]]

Specify a list of batch IDs to pull the data for.

model_id: Optional[str]

The ID of the model.

model_package_id: Optional[str]

The ID of the model package.

segment_attribute: Optional[str]

The name of the segment on which segment analysis is being performed.

segment_value: Optional[str]

The value of the segment_attribute to segment on.

Returns:

custom_metric_summary: CustomMetricBatchSummary

The batch summary of the custom metric.

Return type:

CustomMetricBatchSummary

class datarobot.models.deployment.custom_metrics.HostedCustomMetricTemplate(id, name, description, custom_metric_metadata, default_environment, items, template_metric_type)

Template for hosted custom metric.

classmethod list(search=None, order_by=None, metric_type=None, offset=None, limit=None)

List all hosted custom metric templates.

Parameters:

search: Optional[str]

Search string.

order_by: Optional[ListHostedCustomMetricTemplatesSortQueryParams]

Ordering field.

metric_type: Optional[HostedCustomMetricsTemplateMetricTypeQueryParams]

Type of the metric.

offset: Optional[int]

Offset for pagination.

limit: Optional[int]

Limit for pagination.

Returns:

templates: List[HostedCustomMetricTemplate]

Return type:

List[HostedCustomMetricTemplate]

classmethod get(template_id)

Get a hosted custom metric template by ID.

Parameters:

template_id: str

ID of the template.

Returns:

templateHostedCustomMetricTemplate

Return type:

HostedCustomMetricTemplate

class datarobot.models.deployment.custom_metrics.HostedCustomMetric(id, deployment, units, type, is_model_specific, directionality, time_step, created_at, created_by, name, custom_job_id, description=None, schedule=None, baseline_values=None, timestamp=None, value=None, sample_count=None, batch=None, parameter_overrides=None)

Hosted custom metric.

classmethod list(job_id, skip=None, limit=None)

List all hosted custom metrics for a job.

Parameters:

job_id: str

ID of the job.

Returns:

metrics: List[HostedCustomMetric]

Return type:

List[HostedCustomMetric]

classmethod create_from_template(template_id, deployment_id, job_name, custom_metric_name, job_description=None, custom_metric_description=None, sidecar_deployment_id=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Create a hosted custom metric from a template. A shortcut for 2 calls: Job.from_custom_metric_template(template_id) HostedCustomMetrics.create_from_custom_job()

Parameters:

template_id: str

ID of the template.

deployment_id: str

ID of the deployment.

job_name: str

Name of the job.

custom_metric_name: str

Name of the metric.

job_description: Optional[str]

Description of the job.

custom_metric_description: Optional[str]

Description of the metric.

sidecar_deployment_id: Optional[str]

ID of the sidecar deployment.

baseline_value: Optional[float]

Baseline value.

timestamp: Optional[MetricTimestampSpoofing]

Timestamp details.

value: Optional[ValueField]

Value details.

sample_count: Optional[SampleCountField]

Sample count details.

batch: Optional[BatchField]

Batch details.

schedule: Optional[Schedule]

Schedule details.

parameter_overrides: Optional[List[RuntimeParameterValue]]

Parameter overrides.

Returns:

metric: HostedCustomMetric

Return type:

HostedCustomMetric

classmethod create_from_custom_job(custom_job_id, deployment_id, name, description=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Create a hosted custom metric from existing custom job.

Parameters:

custom_job_id: str

ID of the custom job.

deployment_id: str

ID of the deployment.

name: str

Name of the metric.

description: Optional[str]

Description of the metric.

baseline_value: Optional[float]

Baseline value.

timestamp: Optional[MetricTimestampSpoofing]

Timestamp details.

value: Optional[ValueField]

Value details.

sample_count: Optional[SampleCountField]

Sample count details.

batch: Optional[BatchField]

Batch details.

schedule: Optional[Schedule]

Schedule details.

parameter_overrides: Optional[List[RuntimeParameterValue]]

Parameter overrides.

Returns:

metric: HostedCustomMetric

Return type:

HostedCustomMetric

update(name=None, description=None, units=None, directionality=None, aggregation_type=None, baseline_value=None, timestamp=None, value=None, sample_count=None, batch=None, schedule=None, parameter_overrides=None)

Update the hosted custom metric.

Parameters:

name: Optional[str]

Name of the metric.

description: Optional[str]

Description of the metric.

units: Optional[str]

Units of the metric.

directionality: Optional[str]

Directionality of the metric.

aggregation_type: Optional[CustomMetricAggregationType]

Aggregation type of the metric.

baseline_value: Optional[float]

Baseline values.

timestamp: Optional[MetricTimestampSpoofing]

Timestamp details.

value: Optional[ValueField]

Value details.

sample_count: Optional[SampleCountField]

Sample count details.

batch: Optional[BatchField]

Batch details.

schedule: Optional[Schedule]

Schedule details.

parameter_overrides: Optional[List[RuntimeParameterValue]]

Parameter overrides.

Returns:

metric: HostedCustomMetric

Return type:

HostedCustomMetric

delete()

Delete the hosted custom metric.

Return type:

None

class datarobot.models.deployment.custom_metrics.DeploymentDetails(id, name, creator_first_name=None, creator_last_name=None, creator_username=None, creator_gravatar_hash=None, created_at=None)

Information about a hosted custom metric deployment.

class datarobot.models.deployment.custom_metrics.MetricBaselineValue(value)

The baseline values for a custom metric.

class datarobot.models.deployment.custom_metrics.SampleCountField(column_name)

A weight column used with columnar datasets if pre-aggregated metric values are provided.

class datarobot.models.deployment.custom_metrics.ValueField(column_name)

A custom metric value source for when reading values from a columnar dataset like a file.

class datarobot.models.deployment.custom_metrics.MetricTimestampSpoofing(column_name, time_format=None)

Custom metric timestamp spoofing. Occurs when reading values from a file, like a dataset. By default, replicates pd.to_datetime formatting behavior.

class datarobot.models.deployment.custom_metrics.BatchField(column_name)

A custom metric batch ID source for when reading values from a columnar dataset like a file.

class datarobot.models.deployment.custom_metrics.HostedCustomMetricBlueprint(id, directionality, units, type, time_step, is_model_specific, custom_job_id, created_at, updated_at, created_by, updated_by, categories=None)

Hosted custom metric blueprints provide an option to share custom metric settings between multiple custom metrics sharing the same custom jobs. When a custom job of a hosted custom metric type is connected to the deployment, all the custom metric parameters from the blueprint are automatically copied.

classmethod get(custom_job_id)

Get a hosted custom metric blueprint.

Parameters:

custom_job_id: str

ID of the custom job.

Returns:

blueprint: HostedCustomMetricBlueprint

Return type:

HostedCustomMetricBlueprint

classmethod create(custom_job_id, directionality, units, type, time_step, is_model_specific)

Create a hosted custom metric blueprint.

Parameters:

custom_job_id: str

ID of the custom job.

directionality: str

Directionality of the metric.

units: str

Units of the metric.

type: str

Type of the metric.

time_step: str

Time step of the metric.

is_model_specific: bool

Whether the metric is model specific.

Returns:

blueprint: HostedCustomMetricBlueprint

Return type:

HostedCustomMetricBlueprint

update(directionality=None, units=None, type=None, time_step=None, is_model_specific=None)

Update a hosted custom metric blueprint.

Parameters:

directionality: Optional[str]

Directionality of the metric.

units: Optional[str]

Units of the metric.

type: Optional[str]

Type of the metric.

time_step: Optional[str]

Time step of the metric.

is_model_specific: Optional[bool]

Determines whether the metric is model specific.

Returns:

updated_blueprint: HostedCustomMetricBlueprint

Return type:

HostedCustomMetricBlueprint