Custom Metrics

Custom metrics are used to compute and monitor user-defined metrics.

Manage custom metrics

Use the following commands to manage custom metrics:

Create custom metric

To create a custom metric, use CustomMetric.create, as shown in the following example:

Fill in all metric required custom metric fields:

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="My custom metric",
    units="x",
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER,
)

Set the baseline value during metric creation:

from datarobot.models.deployment import CustomMetric
from datarobot.enums import (
    CustomMetricAggregationType,
    CustomMetricDirectionality,
    CustomMetricBucketTimeStep,
)

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="My custom metric 2",
    units="y",
    baseline_value=12,
    is_model_specific=True,
    aggregation_type=CustomMetricAggregationType.AVERAGE,
    directionality=CustomMetricDirectionality.HIGHER_IS_BETTER,
    time_step=CustomMetricBucketTimeStep.HOUR,
)

Define the names of the columns that will be used when submitting values from a dataset:

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="My custom metric 3",
    units="z",
    baseline_value=1000,
    is_model_specific=False,
    aggregation_type=CustomMetricAggregationType.SUM,
    directionality=CustomMetricDirectionality.LOWER_IS_BETTER,
    timestamp_column_name="My Timestamp column",
    timestamp_format="%d/%m/%y",
    value_column_name="My Value column",
    sample_count_column_name="My Sample Count column",
)

For batches:

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.create(
    deployment_id="5c939e08962d741e34f609f0",
    name="My custom metric 4",
    units="z",
    baseline_value=1000,
    is_model_specific=False,
    aggregation_type=CustomMetricAggregationType.SUM,
    directionality=CustomMetricDirectionality.LOWER_IS_BETTER,
    batch_column_name="My Batch column",
)

List custom metrics

To list all custom metrics available for a given deployment, use CustomMetric.list, as in the following example:

from datarobot.models.deployment import CustomMetric

custom_metrics = CustomMetric.list(deployment_id="5c939e08962d741e34f609f0")

custom_metrics
>>> [CustomMetric('66015bdda7ba87e66baa09ee' | 'My custom metric 2'),
     CustomMetric('66015bdc5f850c5df3aa09f0' | 'My custom metric')]

Retrieve custom metrics

To get a custom metric by unique identifier, use CustomMetric.get, as in the following example:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

custom_metric
>>> CustomMetric('66015bdc5f850c5df3aa09f0' | 'My custom metric')

Update custom metrics

To get a custom metric by unique identifier and update it, use CustomMetric.get() and then update(), as in the following example:

from datarobot.models.deployment import CustomMetric
from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

custom_metric.update(
    name="Updated custom metric",
    units="foo",
    baseline_value=-12,
    aggregation_type=CustomMetricAggregationType.SUM,
    directionality=CustomMetricDirectionality.LOWER_IS_BETTER,
)

Unset custom metric baseline

To reset the current metric baseline, use a separate method unset_baseline(), as in the following example:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

custom_metric.baseline_values
>>> [{'value': -12.0}]

custom_metric.unset_baseline()
custom_metric.baseline_values
>>> []

Delete custom metrics

To delete a custom metric by unique identifier, use CustomMetric.delete, as in the following example:

from datarobot.models.deployment import CustomMetric

CustomMetric.delete(deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

Submit custom metric values

Use the following commands to submit custom metric values:

Submit values from JSON

To submit aggregated custom metric values from JSON, use submit_values method, as shown in the following example:

Submit data in the form of a list of dictionaries:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

data = [{'value': 12, 'sample_size': 3, 'timestamp': '2024-03-15T18:00:00'},
        {'value': 11, 'sample_size': 5, 'timestamp': '2024-03-15T17:00:00'},
        {'value': 14, 'sample_size': 3, 'timestamp': '2024-03-15T16:00:00'}]

custom_metric.submit_values(data=data)

# data witch association IDs
data = [{'value': 15, 'sample_size': 2, 'timestamp': '2024-03-15T21:00:00', 'association_id': '65f44d04dbe192b552e752aa'},
        {'value': 13, 'sample_size': 6, 'timestamp': '2024-03-15T20:00:00', 'association_id': '65f44d04dbe192b552e753bb'},
        {'value': 17, 'sample_size': 2, 'timestamp': '2024-03-15T19:00:00', 'association_id': '65f44d04dbe192b552e754cc'}]

custom_metric.submit_values(data=data)

Submit data in the form of pandas DataFrame:

from datetime import datetime
import pandas as pd
from datarobot.models.deployment import CustomMetric

df = pd.DataFrame(
    data={
        "timestamp": [
            datetime(year=2024, month=3, day=10),
            datetime(year=2024, month=3, day=11),
            datetime(year=2024, month=3, day=12),
            datetime(year=2024, month=3, day=13),
            datetime(year=2024, month=3, day=14),
            datetime(year=2024, month=3, day=15),
        ],
        "value": [28, 34, 29, 1, 2, 13],
        "sample_size": [1, 2, 3, 4, 1, 2],
    }
)
custom_metric.submit_values(data=df)

For deployment-specific metrics, do not provide model information. For model specific metrics set model_package_id or model_id:

custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25")

custom_metric.submit_values(data=data, model_id="6444482e5583f6ee2e572265")

Use a dry run for test uploads (without saving metric data on the DR side), this option is disabled by default:

custom_metric.submit_values(data=data, dry_run=True)

To send data for a given segment, it must be specified as follow, more than one segment can be specified:

segments = [{"name": "custom_seg", "value": "baz"}]
custom_metric.submit_values(data=data, segments=segments)

Batch mode requires specifying batch IDs, for batches always specify a model by model_package_id or model_id:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f600e1", custom_metric_id="65f17bdcd2d66683cdfc2224")

data = [{'value': 12, 'sample_size': 3, 'batch': '65f44c93fedc5de16b673aaa'},
        {'value': 11, 'sample_size': 5, 'batch': '65f44c93fedc5de16b673bbb'},
        {'value': 14, 'sample_size': 3, 'batch': '65f44c93fedc5de16b673ccc'}]

custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25")

Submit a single value

To report a single metric value at the current moment, use submit_single_value method, as in the following example:

For deployment-specific metrics:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

custom_metric.submit_single_value(value=16)

For model specific metrics set model_package_id or model_id:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

custom_metric.submit_single_value(value=16, model_package_id="6421df32525c58cc6f991f25")

custom_metric.submit_single_value(value=16, model_id="6444482e5583f6ee2e572265")

Dry run and segments work analogously to reporting aggregated metric values:

custom_metric.submit_single_value(value=16, dry_run=True)

segments = [{"name": "custom_seg", "value": "boo"}]
custom_metric.submit_single_value(value=16, segments=segments)

The sent value timestamp indicates the time the request was sent, the number of samples values is always 1. This method does not support batch submissions.

Submit values from a dataset

To report aggregated custom metrics values from dataset (AI catalog), use submit_values_from_catalog method, as in the following example:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

# for deployment specific metrics
custom_metric.submit_values_from_catalog(dataset_id="61093144cabd630828bca321")

# for model specific metrics set model_package_id or model_id
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25"
)

For segmented analysis define the name of the column in the dataset and the segment it corresponds to:

segments = [{"name": "custom_seg", "column": "column_with_segment_values"}]
custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca321",
    model_package_id="6421df32525c58cc6f991f25",
    segments=segments
)

For batches, specify batch IDs in the dataset or send the entire dataset for a single batch ID:

custom_metric.submit_values_from_catalog(
    dataset_id="61093144cabd630828bca432",
    model_package_id="6421df32525c58cc6f991f25",
    batch_id="65f7f71198c2f234b4cb2f7d"
)

The names of the columns in the dataset should correspond to the names of the columns that were defined in the custom metric. In addition, the format of the timestamps should also be the same as defined in the metric. If the sample size is not specified, it is treated as a 1 sample by default. The following is an example of the shape of a dataset saved in the AI catalog:

timestamp	sample_size	value
12/12/22	1	22
13/12/22	2	23
14/12/22	3	24
15/12/22	4	25

Sample dataset for batches:

batch	sample_size	value
6572db2c9f9d4ad3b9de33d0	1	22
6572db2c9f9d4ad3b9de33d0	2	23
6572db319f9d4ad3b9de33d9	3	24
6572db319f9d4ad3b9de33d9	4	25

Retrieve custom metric values over time

Use the following commands to retrieve custom metric values:

Retrieve values over a time period

To retrieve values of a custom metric over a time period, use get_values_over_time, as in the following example:

from datetime import datetime, timedelta
from datarobot.enums import BUCKET_SIZE
from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

now = datetime.now()
# specify the time window and bucket size by which results are grouped, the default bucket is 7 days
values_over_time = custom_metric.get_values_over_time(
    start=now - timedelta(days=2), end=now, bucket_size=BUCKET_SIZE.P1D)

values_over_time
>>> CustomMetricValuesOverTime('2024-03-21 20:15:00+00:00'- '2024-03-23 20:15:00+00:00')"

values_over_time.bucket_values
>>>{datetime.datetime(2024, 3, 22, 10, 0, tzinfo=tzutc()): 1.0,
>>> datetime.datetime(2024, 3, 22, 11, 0, tzinfo=tzutc()): 123.0}}

values_over_time.bucket_sample_sizes
>>>{datetime.datetime(2024, 3, 22, 10, 0, tzinfo=tzutc()): 1,
>>> datetime.datetime(2024, 3, 22, 11, 0, tzinfo=tzutc()): 1}}

values_over_time.get_buckets_as_dataframe()
>>>                        start                       end  value  sample_size
>>> 0  2024-03-21 00:00:00+00:00 2024-03-22 00:00:00+00:00    1.0            1
>>> 1  2024-03-22 00:00:00+00:00 2024-03-23 00:00:00+00:00  123.0            1

For model specific metrics set model_package_id or model_id:

values_over_time = custom_metric.get_values_over_time(
    start=now - timedelta(days=1), end=now, model_package_id="6421df32525c58cc6f991f25")

values_over_time = custom_metric.get_values_over_time(
    start=now - timedelta(days=1), end=now, model_id="6444482e5583f6ee2e572265")

To retrieve values for a specific segment, specify the segment name and its value:

values_over_time = custom_metric.get_values_over_time(
    start=now - timedelta(days=1), end=now, segment_attribute="custom_seg", segment_value="val_1")

Retrieve a summary over a time period

To retrieve summary of a custom metric over a time period, use get_summary method, as in the following example:

from datetime import datetime, timedelta
from datarobot.enums import BUCKET_SIZE
from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113")

now = datetime.now()
# specify the time window
summary = custom_metric.get_summary(start=now - timedelta(days=7), end=now)

print(summary)
>> "CustomMetricSummary(2024-03-15 15:52:13.392178+00:00 - 2024-03-22 15:52:13.392168+00:00:
{'id': '65fd9b1c0c1a840bc6751ce0', 'name': 'My custom metric', 'value': 215.0, 'sample_count': 13,
'baseline_value': 12.0, 'percent_change': 24.02})"

For model specific metrics set model_package_id or model_id:

summary = custom_metric.get_summary(
    start=now - timedelta(days=7), end=now, model_package_id="6421df32525c58cc6f991f25")

summary = custom_metric.get_summary(
    start=now - timedelta(days=7), end=now, model_id="6444482e5583f6ee2e572265")

To retrieve summary for a specific segment, specify the segment name and its value:

summary = custom_metric.get_summary(
    start=now - timedelta(days=7), end=now, segment_attribute="custom_seg", segment_value="val_1")

Retrieve values over batch

To retrieve values of a custom metric over batch, use get_values_over_batch, as in the following example:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
# all batch metrics all model specific
values_over_batch = custom_metric.get_values_over_batch(model_package_id='6421df32525c58cc6f991f25')

values_over_batch.bucket_values
>>> {'6572db2c9f9d4ad3b9de33d0': 35.0, '6572db2c9f9d4ad3b9de44e1': 105.0}

values_over_batch.bucket_sample_sizes
>>> {'6572db2c9f9d4ad3b9de33d0': 6, '6572db2c9f9d4ad3b9de44e1': 8}

values_over_batch.get_buckets_as_dataframe()
>>>                    batch_id                     batch_name  value  sample_size
>>> 0  6572db2c9f9d4ad3b9de33d0  Batch 1 - 03/26/2024 13:04:46   35.0            6
>>> 1  6572db2c9f9d4ad3b9de44e1  Batch 2 - 03/26/2024 13:06:04  105.0            8

For specific batches, set batch_ids:

values_over_batch = custom_metric.get_values_over_batch(
    model_package_id='6421df32525c58cc6f991f25', batch_ids=["65f44c93fedc5de16b673aaa", "65f44c93fedc5de16b673bbb"])

To retrieve values for a specific segment, specify the segment name and its value:

values_over_batch = custom_metric.get_values_over_batch(
    model_package_id='6421df32525c58cc6f991f25', segment_attribute="custom_seg", segment_value="val_1")

Retrieve a summary over batch

To retrieve summary of a custom metric over batch, use get_summary, as in the following example:

from datarobot.models.deployment import CustomMetric

custom_metric = CustomMetric.get(
    deployment_id="5c939e08962d741e34f609f0",
    custom_metric_id="65f17bdcd2d66683cdfc1113"
)
# all batch metrics all model specific
batch_summary = custom_metric.get_batch_summary(model_package_id='6421df32525c58cc6f991f25')

print(batch_summary)
>> CustomMetricBatchSummary({'id': '6605396413434b3a7b74342c', 'name': 'batch metric', 'value': 41.25,
'sample_count': 28, 'baseline_value': 123.0, 'percent_change': -66.46})

For specific batches, set batch_ids:

batch_summary = custom_metric.get_batch_summary(
    model_package_id='6421df32525c58cc6f991f25', batch_ids=["65f44c93fedc5de16b673aaa", "65f44c93fedc5de16b673bbb"])

To retrieve values for a specific segment, specify the segment name and its value:

batch_summary = custom_metric.get_batch_summary(
    model_package_id='6421df32525c58cc6f991f25', segment_attribute="custom_seg", segment_value="val_1")