Custom Metrics

Hosted custom metrics run user-provided code on DataRobot infrastructure to calculate your organization’s customized metrics. DataRobot provides a variety of templates for common metrics. These metrics can be used as is, or as a starting point for user-provided metrics. In this tutorial, you create a hosted custom metric using the Python SDK.

Prerequisites

Before you start, import all objects used in this tutorial and initialize the DataRobot client:

import datarobot as dr
from datarobot.enums import HostedCustomMetricsTemplateMetricTypeQueryParams
from datarobot.models.deployment.custom_metrics import HostedCustomMetricTemplate, HostedCustomMetric, \
    HostedCustomMetricBlueprint, CustomMetric, MetricTimestampSpoofing, ValueField, SampleCountField, BatchField
from datarobot.models.registry import JobRun
from datarobot.models.registry.job import Job
from datarobot import Deployment
from datarobot.models.runtime_parameters import RuntimeParameterValue
from datarobot.models.types import Schedule

dr.Client(token="<DataRobot API Token>", endpoint="<DataRobot URL>")
gen_ai_deployment_1 = Deployment.get('<Deployment Id>')

List hosted custom metrics templates

Before creating a hosted custom metric from a template, retrieve the LLM metric template to use as the basis of the new metric. To do this, specify the metric_type and, because the deployments are LLM models handling Japanese text, search for the specific metric by name, limiting the search to 1 result. Store the result in templates for the next step.

templates = HostedCustomMetricTemplate.list(
    search="[JP] Character Count",
    metric_type=HostedCustomMetricsTemplateMetricTypeQueryParams.LLM,
    limit=1,
    offset=0,
)

Create hosted custom metric in single step

After locating the custom metric template, create the hosted custom metric from that template. This method is a shortcut, combining two steps to create the new custom metric from the retrieved template: 1. Create a custom job for a hosted custom metric from the template retrieved in the previous step (stored in templates). 2. Connect the hosted custom metric job to the deployment defined during the prerequisites step (stored in gen_ai_deployment_1). Because we are creating 2 objects, specify both the job name and custom metric name in addition to the template and deployment IDs. Additionally, define the job schedule and the runtime parameter overrides for the deployment.

hosted_custom_metric = HostedCustomMetric.create_from_template(
    template_id=templates[0].id,
    deployment_id=gen_ai_deployment_1.id,
    job_name="Hosted Custom Metric Character Count",
    custom_metric_name="Character Count",
    job_description="Hosted Custom Metric",
    custom_metric_description="LLM Character Count",
    baseline_value=10,
    timestamp=MetricTimestampSpoofing(
        column_name="timestamp",
        time_format="%Y-%m-%d %H:%M:%S",
    ),
    value = ValueField(column_name="value"),
    sample_count=SampleCountField(column_name='Sample Count'),
    batch=BatchField(column_name='Batch'),
    schedule=Schedule(
        day_of_week=[0],
        hour=['*'],
        minute=['*'],
        day_of_month=[12],
        month=[1],
    ),
    parameter_overrides=[RuntimeParameterValue(field_name='DRY_RUN', value="0", type="string")]
)

Once we have created the hosted custom metric, you can initiate the manual run.

job_run = JobRun.create(
        job_id=hosted_custom_metric.custom_job_id
        runtime_parameter_values=[
            RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"),
            RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"),
            RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=hosted_custom_metric.id, type="customMetric"),
        ]
    )
    print(job_run.status)

Create hosted custom metric in two steps

You can also perform these steps manually, in sequence. This is useful if you want to edit the custom metric blueprint before attaching the custom job to the deployment. When you attach the job to the deployment, most settings are copied from blueprint (unless you provide an override) . To create the hosted custom metric manually, first, create a custom job from the template retrieved earlier (stored in templates).

job = Job.create_from_custom_metric_gallery_template(
    template_id=templates[0].id,
    name="Job created from template",
    description="Job created from template"
)

Next, retrieve the default blueprint, provided by the template, and edit it.

blueprint = HostedCustomMetricBlueprint.get(job.id)
print(f"Original directionality: {blueprint.directionality}")

Then, update the parameters of the custom metric in the blueprint.

updated_blueprint = blueprint.update(
    directionality='lowerIsBetter',
    units='characters',
    type='gauge',
    time_step='hour',
    is_model_specific=False
)
print(f"Updated directionality: {updated_blueprint.directionality}")

Now create the hosted custom metric. As in the shortcut method, you can provide the job schedule, runtime parameter overrides, and custom metric parameters specific to this deployment.

another_hosted_custom_metric = HostedCustomMetric.create_from_custom_job(
    custom_job_id=job.id,
    deployment_id=gen_ai_deployment_1.id,
    name="Custom metric created in 2 steps",
)

After creating and configuring the metric, verify that the changes to the blueprint are reflected in the custom metric.

another_custom_metric = CustomMetric.get(custom_metric_id=another_hosted_custom_metric.id, deployment_id=gen_ai_deployment_1.id)
print(f"Directionality of another custom metric: {another_custom_metric.directionality}")

Finally, create a manual job run for the custom metric job.

job_run = JobRun.create(
    job_id=job.id,
    runtime_parameter_values=[
        RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"),
        RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"),
        RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=another_hosted_custom_metric.id, type="customMetric"),
    ]
)
print(job_run.status)

List Hosted Custom Metrics

To list all hosted custom metrics associated with a custom job, use the following code:

hosted_custom_metrics = HostedCustomMetric.list(deployment_id=hosted_custom_metric.custom_job_id)
for metric in hosted_custom_metrics:
    print(metric.name)

Delete Hosted Custom Metrics

In addition, you can delete the hosted custom metric, removing the custom metric from deployment but keeping the job, allowing you to create the metric for another deployment.

hosted_custom_metric.delete()
another_hosted_custom_metric.delete()

If necessary, you can delete the entire custom job. If there are any custom metrics associated with that job, they are also deleted.

job.delete()