Custom models

Custom models provide the ability to run arbitrary modeling code in a user-defined environment.

Manage execution environments

The execution environment defines the runtime environment for custom models. Execution environment version is a revision of execution environment with an actual runtime definition. Refer to DataRobot User Models repository for sample environments.

Create an execution environment

To create an execution environment:

import datarobot as dr

execution_environment = dr.ExecutionEnvironment.create(
    name="Python3 PyTorch Environment",
    description="This environment contains Python3 pytorch library.",
)

execution_environment.id
>>> '5b6b2315ca36c0108fc5d41b'

Create an execution environment version from Docker Context or Docker URI

You can create an Execution Environment Version using either a Docker image URI, a Docker context, or both. If you provide both, the environment version is built from the image URI, while the context is uploaded for informational purposes and can be later downloaded.

There are two ways to create an execution environment version: synchronously and asynchronously.

The synchronous method blocks program execution until the execution environment version is created or creation fails.

import datarobot as dr

# Use the execution_environment created previously

environment_version = dr.ExecutionEnvironmentVersion.create(
    execution_environment.id,
    docker_context_path="datarobot-user-models/public_dropin_environments/python3_pytorch",
    max_wait=3600,  # 1 hour timeout
)

environment_version.id
>>> '5eb538959bc057003b487b2d'
environment_version.build_status
>>> 'success'

The asynchronous method does not block execution, but the execution environment version will not be ready for use until the creation process is finished. In this case, you must manually call refresh() for the execution environment version and check if its build_status is “success”. To create an execution environment version without blocking a program, set max_wait to None:

import datarobot as dr

# Use the execution environment created earlier

# create environment version from docker context
environment_version = dr.ExecutionEnvironmentVersion.create(
    execution_environment.id,
    docker_context_path="datarobot-user-models/public_dropin_environments/python3_pytorch",
    max_wait=None,  # Set None to not block execution on this method
)

environment_version.id
>>> '5eb538959bc057003b487b2d'
environment_version.build_status
>>> 'processing'

# After some time
environment_version.refresh()
environment_version.build_status
>>> 'success'

# now create anoter environment version from docker image URI
environment_version = dr.ExecutionEnvironmentVersion.create(
    execution_environment.id,
    docker_image_uri="test_org/test_repo:test_tag",
    max_wait=None,  # set None to not block execution on this method
)

environment_version.id
>>> '5eb538959bc057003b4943d2'
environment_version.build_status
>>> 'success'
environment_version.docker_image_uri
'test_org/test_repo:test_tag'

If your environment requires additional metadata to be supplied for models using it, you can create an environment with additional metadata keys. Custom model versions that use this environment must specify values for these keys before they can be used to run tests or make deployments. The values will be baked in as environment variables with field_name as the environment variable name.

import datarobot as dr
from datarobot.models.execution_environment import RequiredMetadataKey

execution_environment = dr.ExecutionEnvironment.create(
    name="Python3 PyTorch Environment",
    description="This environment contains Python3 pytorch library.",
    required_metadata_keys=[
        RequiredMetadataKey(field_name="MY_VAR", display_name="A value needed by hte environment")
    ],
)

model_version = dr.CustomModelVersion.create_clean(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    folder_path=custom_model_folder,
    required_metadata={"MY_VAR": "a value"}
)

List execution environments

To list execution environments available to the user:

import datarobot as dr

execution_environments = dr.ExecutionEnvironment.list()
execution_environments
>>> [ExecutionEnvironment('[DataRobot] Python 3 PyTorch Drop-In'), ExecutionEnvironment('[DataRobot] Java Drop-In')]

environment_versions = dr.ExecutionEnvironmentVersion.list(execution_environment.id)
environment_versions
>>> [ExecutionEnvironmentVersion('v1')]

Refer to ExecutionEnvironment for properties of the execution environment object and ExecutionEnvironmentVersion for properties of the execution environment object version.

You can also filter the execution environments that are returned by passing a string as a search_for parameter. Only the execution environments that contain the passed string in name or description are returned.

import datarobot as dr

execution_environments = dr.ExecutionEnvironment.list(search_for='java')
execution_environments
>>> [ExecutionEnvironment('[DataRobot] Java Drop-In')]

Execution environment versions can be filtered by build status.

import datarobot as dr

environment_versions = dr.ExecutionEnvironmentVersion.list(
    execution_environment.id, dr.EXECUTION_ENVIRONMENT_VERSION_BUILD_STATUS.PROCESSING
)
environment_versions
>>> [ExecutionEnvironmentVersion('v1')]

Retrieve an execution environment

To retrieve an execution environment and an execution environment version by identifier (rather than list all available environments):

import datarobot as dr

execution_environment = dr.ExecutionEnvironment.get(execution_environment_id='5506fcd38bd88f5953219da0')
execution_environment
>>> ExecutionEnvironment('[DataRobot] Python 3 PyTorch Drop-In')

environment_version = dr.ExecutionEnvironmentVersion.get(
    execution_environment_id=execution_environment.id, version_id='5eb538959bc057003b487b2d')
environment_version
>>> ExecutionEnvironmentVersion('v1')

Update an execution environment

To update name or description of the execution environment:

import datarobot as dr

execution_environment = dr.ExecutionEnvironment.get(execution_environment_id='5506fcd38bd88f5953219da0')
execution_environment.update(name='new name', description='new description')

Delete Execution Environment

To delete the execution environment and execution environment version:

import datarobot as dr

execution_environment = dr.ExecutionEnvironment.get(execution_environment_id='5506fcd38bd88f5953219da0')
execution_environment.delete()

Get execution environment build logs

To get an execution environment version build log:

import datarobot as dr

environment_version = dr.ExecutionEnvironmentVersion.get(
    execution_environment_id='5506fcd38bd88f5953219da0', version_id='5eb538959bc057003b487b2d')
log, error = environment_version.get_build_log()

Manage custom models

A custom model is user-defined modeling code that supports making predictions against it. Custom models support regression and binary classification target types. To upload actual modeling code, you must create a custom model version for a custom model. See Custom Model Version documentation for more information.

Create a custom model

Regression model

To create a regression custom model:

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 PyTorch Custom Model',
    target_type=dr.TARGET_TYPE.REGRESSION,
    target_name='MEDV',
    description='This is a Python3-based custom model. It has a simple PyTorch model built on boston housing',
    language='python'
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

Binary classification model

When creating a binary classification custom model, positive_class_label and negative_class_label must be set:

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 PyTorch Custom Model',
    target_type=dr.TARGET_TYPE.BINARY,
    target_name='readmitted',
    positive_class_label='False',
    negative_class_label='True',
    description='This is a Python3-based custom model. It has a simple PyTorch model built on 10k_diabetes dataset',
    language='Python 3'
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

Multiclass model

When creating a multiclass classification custom model, you must provide class_labels:

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 PyTorch Custom Model',
    target_type=dr.TARGET_TYPE.MULTICLASS,
    target_name='readmitted',
    class_labels=['hot dog', 'burrito', 'hoagie', 'reuben'],
    description='This is a Python3-based custom model. It has a simple PyTorch model built on sandwich dataset',
    language='Python 3'
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

Multiclass labels can also be provided as a file in cases where there are many class labels. The file should have each class label separated by a new line.

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 PyTorch Custom Model',
    target_type=dr.TARGET_TYPE.MULTICLASS,
    target_name='readmitted',
    class_labels_file='/path/to/classlabels.txt',
    description='This is a Python3-based custom model. It has a simple PyTorch model built on sandwich dataset',
    language='Python 3'
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

Unstructured model

For unstructured models, the target_name parameter is optional and ignored if provided. To create an unstructured custom model:

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 Unstructured Custom Model',
    target_type=dr.TARGET_TYPE.UNSTRUCTURED,
    description='This is a Python3-based unstructured model',
    language='python'
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

Anomaly detection model

For anomaly detection models, the target_name parameter is also optional and is ignored if provided. To create an anomaly detection custom model:

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 Unstructured Custom Model',
    target_type=dr.TARGET_TYPE.ANOMALY,
    description='This is a Python3-based anomaly detection model',
    language='python'
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

k8s resources

Custom model k8s resources are optional and unless specifically provided, the configured defaults are used.

To create a custom model with specific k8s resources:

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 PyTorch Custom Model',
    target_type=dr.TARGET_TYPE.BINARY,
    target_name='readmitted',
    positive_class_label='False',
    negative_class_label='True',
    description='This is a Python3-based custom model. It has a simple PyTorch model built on 10k_diabetes dataset',
    language='Python 3',
    maximum_memory=512*1024*1024,
)

Assign training data to custom models

To create a custom model that enables training data assignment on the model version level, provide the is_training_data_for_versions_permanently_enabled=True parameter. For more information, refer to the Custom model version creation with training data documentation.

import datarobot as dr

custom_model = dr.CustomInferenceModel.create(
    name='Python 3 PyTorch Custom Model',
    target_type=dr.TARGET_TYPE.REGRESSION,
    target_name='MEDV',
    description='This is a Python3-based custom model. It has a simple PyTorch model built on boston housing',
    language='python',
    is_training_data_for_versions_permanently_enabled=True
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

List custom models

To list the custom models available to you:

import datarobot as dr

dr.CustomInferenceModel.list()
>>> [CustomInferenceModel('my model 2'), CustomInferenceModel('my model 1')]

# use these parameters to filter results:
dr.CustomInferenceModel.list(
    is_deployed=True,  # set to return only deployed models
    order_by='-updated',  # set to define order of returned results
    search_for='model 1',  # return only models containing 'model 1' in name or description
)
>>> CustomInferenceModel('my model 1')

Refer to list() for detailed parameter descriptions.

Retrieve a custom model

To retrieve a specific custom model:

import datarobot as dr

dr.CustomInferenceModel.get('5ebe95044024035cc6a65602')
>>> CustomInferenceModel('my model 1')

Update custom model

To update custom model properties:

import datarobot as dr

custom_model = dr.CustomInferenceModel.get('5ebe95044024035cc6a65602')

custom_model.update(
    name='new name',
    description='new description',
)

Please, refer to update() for the full list of properties that can be updated.

Download latest revision of a custom model

To download content of the latest Custom Model Version of CustomInferenceModel as a ZIP archive:

import datarobot as dr

path_to_download = '/home/user/Documents/myModel.zip'

custom_model = dr.CustomInferenceModel.get('5ebe96b84024035cc6a6560b')

custom_model.download_latest_version(path_to_download)

Assign training data to a custom model

This example assigns training data on the model level. To assign training data on the model version level, see the Custom model version creation with training data documentation.

To assign training data to custom inference model:

import datarobot as dr

path_to_dataset = '/home/user/Documents/trainingDataset.csv'
dataset = dr.Dataset.create_from_file(file_path=path_to_dataset)

custom_model = dr.CustomInferenceModel.get('5ebe96b84024035cc6a6560b')

custom_model.assign_training_data(dataset.id)

To assign training data without blocking a program, set max_wait to None:

import datarobot as dr

path_to_dataset = '/home/user/Documents/trainingDataset.csv'
dataset = dr.Dataset.create_from_file(file_path=path_to_dataset)

custom_model = dr.CustomInferenceModel.get('5ebe96b84024035cc6a6560b')

custom_model.assign_training_data(
    dataset.id,
    max_wait=None
)

custom_model.training_data_assignment_in_progress
>>> True

# after some time
custom_model.refresh()
custom_model.training_data_assignment_in_progress
>>> False

Note: training data must be assigned to retrieve feature impact from a custom model version. See the Custom Model Version documentation.

Manage versions

Modeling code for custom models can be uploaded by creating a custom model version. When creating a Custom Model Version, the version must be associated with a base execution environment. If the base environment supports additional model dependencies (R or Python environments) and the custom model version contains a valid requirements.txt file, the model version will run in an environment based on the base environment with the additional dependencies installed.

Create a custom model version

You can upload custom model content by creating a clean custom model version:

import os
import datarobot as dr

custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch"

# Add files from the folder to the custom model
model_version = dr.CustomModelVersion.create_clean(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    folder_path=custom_model_folder,
)

custom_model.id
>>> '5b6b2315ca36c0108fc5d41b'

# Alternatively, add a list of files to the custom model
model_version_2 = dr.CustomModelVersion.create_clean(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    files=[(os.path.join(custom_model_folder, 'custom.py'), 'custom.py')],
)

# You can also set k8s resources to the custom model
model_version_3 = dr.CustomModelVersion.create_clean(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    files=[(os.path.join(custom_model_folder, 'custom.py'), 'custom.py')],
    network_egress_policy=dr.NETWORK_EGRESS_POLICY.PUBLIC,
    maximum_memory=512*1024*1024,
    replicas=1,
)

To create a new custom model version from a previous one that modifies some files, use the following code.

import os
import datarobot as dr

custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch"

file_to_delete = model_version_2.items[0].id

model_version_3 = dr.CustomModelVersion.create_from_previous(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    files=[(os.path.join(custom_model_folder, 'custom.py'), 'custom.py')],
    files_to_delete=[file_to_delete],
)

Reference CustomModelFileItem for more information about custom model file properties.

You can specify a custom environment version when creating a custom model version. By default a version of the same environment does not change between consecutive model versions.

However, this behavior can be overridden:

import os
import datarobot as dr

custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch"

# Create a clean version and specify an explicit environment version.
model_version = dr.CustomModelVersion.create_clean(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    base_environment_version_id="642209acc5638929a9b8dc3d",
    folder_path=custom_model_folder,
)

# Create a version from a previous one, specifying an explicit environment version.
model_version_2 = dr.CustomModelVersion.create_from_previous(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    base_environment_version_id="660186775d016eabb290aee9",
)

To create a new custom model version from a previous one, with just new k8s resource values:

import os
import datarobot as dr

custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch"

file_to_delete = model_version_2.items[0].id

model_version_3 = dr.CustomModelVersion.create_from_previous(
    custom_model_id=custom_model.id,
    base_environment_id=execution_environment.id,
    maximum_memory=1024*1024*1024,
)

Create a custom model version with training data

Model version creation allows you to provide training (and holdout) data information. Every custom model has to be explicitly switched to allow training data assignment for model versions. Note that the training data assignment differs for structured and unstructured models, and should be handled differently.

Enable training data assignment for custom model versions

By default, custom model training data is assigned on the model level; for more information, see the Custom model training data assignment documentation. When training data is assigned to a model, the same training data is used for every model version. This method of training data assignment is deprecated and scheduled for removal; however, to avoid introducing issues for existing models, you must individually convert existing models to perform training data assignment by model version.

Note that this change is permanent and cannot be undone. Because the conversion process is irreversible, it is highly recommended that you do not convert critical models to the new training data assignment method. Instead, you should duplicate the existing model and test the new method.

Use the code below to permanently enable a training data assignment on the model version level for the specified model.

import datarobot as dr

dr.Client(token=my_token, endpoint=endpoint)
custom_model = dr.CustomInferenceModel.get(custom_model_id)
custom_model.update(is_training_data_for_versions_permanently_enabled=True)
custom_model.is_training_data_for_versions_permanently_enabled  # True

Assign training data for structured models

Training data assignment is performed asynchronously, so you can create a version in a blocking or non-blocking way (shown in the examples below).

Create a structured model version with blocking (default max_wait=600) and wait for the training data assignment result.

If the training data assignment fails:

• A datarobot.errors.TrainingDataAssignmentError exception is raised. The exception contains the custom model ID, the custom model version ID, and the failure message.

• A new custom model version is still created and can be fetched for further processing, but it’s not possible to create a model package from it or deploy it.

import datarobot as dr
from datarobot.errors import TrainingDataAssignmentError

dr.Client(token=my_token, endpoint=endpoint)

try:
    version = dr.CustomModelVersion.create_from_previous(
        custom_model_id="6444482e5583f6ee2e572265",
        base_environment_id="642209acc563893014a41e24",
        training_dataset_id="6421f2149a4f9b1bec6ad6dd",
    )
except TrainingDataAssignmentError as e:
    print(e)

To fetch the model version in the case of an assignment error:

import datarobot as dr
from datarobot.errors import TrainingDataAssignmentError

dr.Client(token=my_token, endpoint=endpoint)

try:
    version = dr.CustomModelVersion.create_from_previous(
        custom_model_id="6444482e5583f6ee2e572265",
        base_environment_id="642209acc563893014a41e24",
        training_dataset_id="6421f2149a4f9b1bec6ad6dd",
    )
except TrainingDataAssignmentError as e:
    version = CustomModelVersion.get(
        custom_model_id="6444482e5583f6ee2e572265",
        custom_model_version_id=e.custom_model_version_id,
    )
    print(version.training_data.dataset_id)
    print(version.training_data.dataset_version_id)
    print(version.training_data.dataset_name)
    print(version.training_data.assignment_error)

Below is another example of fetching the model version in the case of an assignment error.

import datarobot as dr
from datarobot.errors import TrainingDataAssignmentError

dr.Client(token=my_token, endpoint=endpoint)
custom_model = dr.CustomInferenceModel.get("6444482e5583f6ee2e572265")

try:
    version = dr.CustomModelVersion.create_from_previous(
        custom_model_id="6444482e5583f6ee2e572265",
        base_environment_id="642209acc563893014a41e24",
        training_dataset_id="6421f2149a4f9b1bec6ad6dd",
    )
except TrainingDataAssignmentError as e:
    pass

custom_model.refresh()
version = custom_model.latest_version
print(version.training_data.dataset_id)
print(version.training_data.dataset_version_id)
print(version.training_data.dataset_name)
print(version.training_data.assignment_error)

Create a structured model version with a non-blocking (set max_wat=None) training data assignment.

In this case, it is the user’s responsibility to poll for version.training_data.assignment_in_progress. Once the assignment is finished, check for errors if version.training_data.assignment_in_progress==False. If version.training_data.assignment_error is None, then there is no error.

import datarobot as dr

dr.Client(token=my_token, endpoint=endpoint)

version = dr.CustomModelVersion.create_from_previous(
    custom_model_id="6444482e5583f6ee2e572265",
    base_environment_id="642209acc563893014a41e24",
    training_dataset_id="6421f2149a4f9b1bec6ad6dd",
    max_wait=None,
)

while version.training_data.assignment_in_progress:
    time.sleep(10)
    version.refresh()
if version.training_data.assignment_error:
    print(version.training_data.assignment_error["message"])

Assign training data for unstructured models

For unstructured models, you can provide the parameters training_dataset_id and holdout_dataset_id. The training data assignment is performed synchronously and the max_wait parameter is ignored.

The example below shows how to create an unstructured model version with training and holdout data.

import datarobot as dr

dr.Client(token=my_token, endpoint=endpoint)

version = dr.CustomModelVersion.create_from_previous(
    custom_model_id="6444482e5583f6ee2e572265",
    base_environment_id="642209acc563893014a41e24",
    training_dataset_id="6421f2149a4f9b1bec6ad6dd",
    holdout_dataset_id="6421f2149a4f9b1bec6ad6ef",
)
if version.training_data.assignment_error:
    print(version.training_data.assignment_error["message"])

Remove training data

By default, training and holdout data are copied to a new model version from the previous model version. If you don’t want to keep training and holdout data for the new version, set keep_training_holdout_data to False.

import datarobot as dr

dr.Client(token=my_token, endpoint=endpoint)

version = dr.CustomModelVersion.create_from_previous(
    custom_model_id="6444482e5583f6ee2e572265",
    base_environment_id="642209acc563893014a41e24",
    keep_training_holdout_data=False,
)

List custom model versions

To list custom model versions available to you:

import datarobot as dr

dr.CustomModelVersion.list(custom_model.id)

>>> [CustomModelVersion('v2.0'), CustomModelVersion('v1.0')]

Retrieve a custom model version

To retrieve a specific custom model version, run the code below.

import datarobot as dr

dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b')

>>> CustomModelVersion('v2.0')

Update a version description

To update a custom model version description:

import datarobot as dr

custom_model_version = dr.CustomModelVersion.get(
    custom_model.id,
    custom_model_version_id='5ebe96b84024035cc6a6560b',
)

custom_model_version.update(description='new description')

custom_model_version.description
>>> 'new description'

Download a version

To download the contents of a custom model version as a ZIP archive:

import datarobot as dr

path_to_download = '/home/user/Documents/myModel.zip'

custom_model_version = dr.CustomModelVersion.get(
    custom_model.id,
    custom_model_version_id='5ebe96b84024035cc6a6560b',
)

custom_model_version.download(path_to_download)

Start custom model inference legacy conversion

Custom model versions may include SAS files, with a main program entry point. In order to use a model, a conversion must be run. The conversion can later be fetched and examined by reading the conversion print-outs.

By default, a conversion is initiated in a non-blocking mode. If a max_wait parameter is provided, than the call is blocked until the conversion is completed. The results can than be read by fetching the conversion entity.

import datarobot as dr

    # Read a custom model version
    custom_model_version = dr.CustomModelVersion.get(model_id, model_version_id)

    # Find the main program item ID
    main_program_item_id = None
    for item in cm_ver.items:
            if item.file_name.lower().endswith('.sas'):
                    main_program_item_id = item.id

    # Execute the conversion
    if async:
            # This is a non-blocking call
            conversion_id = dr.models.CustomModelVersionConversion.run_conversion(
                    custom_model_version.custom_model_id,
                    custom_model_version.id,
                    main_program_item_id,
            )
    else:
            # This call is blocked until a completion or a timeout
            conversion_id = dr.models.CustomModelVersionConversion.run_conversion(
                    custom_model_version.custom_model_id,
                    custom_model_version.id,
                    main_program_item_id,
                    max_wait=60,
            )

Monitor model conversion

If a custom model version conversion was initiated in a non-blocking mode, it is possible to monitor the progress as follows:

import datarobot as dr

    while True:
            conversion = dr.models.CustomModelVersionConversion.get(
                    custom_model_id, custom_model_version_id, conversion_id,
            )
            if conversion.conversion_in_progress:
                    logging.info('Conversion is in progress...')
                    time.sleep(1)
            else:
                    if conversion.conversion_succeeded:
                            logging.info('Conversion succeeded')
                    else:
                            logging.error(f'Conversion failed!\n{conversion.log_message}')
                    break

Stop conversion

It is possible to stop a custom model version conversion that is in progress. The call is non-blocking and you may keep monitoring the conversion progress (see above) until is it completed.

import datarobot as dr

    dr.models.CustomModelVersionConversion.stop_conversion(
            custom_model_id, custom_model_version_id, conversion_id,
    )

Calculate Feature Impact

To trigger the calculation of a custom model version’s Feature Impact, training data must be assigned to a custom model. (For more information about custom model training data, reference the custom model documentation.) If training data is assigned, run the following code to trigger the calculation of its Feature Impact:

import datarobot as dr

version = dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b')

version.calculate_feature_impact()

To trigger Feature Impact calculation without blocking a program, set max_wait to None:

import datarobot as dr

version = dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b')

version.calculate_feature_impact(max_wait=None)

Retrieve custom model image Feature Impact

To retrieve a custom model image’s Feature Impact, it must be calculated beforehand. Reference the Custom model version Feature Impact documentation for more information.

To get Feature Impact:

import datarobot as dr

version = dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b')

version.get_feature_impact()
>>> [{'featureName': 'B', 'impactNormalized': 1.0, 'impactUnnormalized': 1.1085356209402688, 'redundantWith': 'B'}...]

Prepare a custom model version for use

If your custom model version has dependencies, a dependency build must be completed before the model can be used. The dependency build installs your model’s dependencies into the base environment associated with the model version.

Start a dependency build

To start a custom model version dependency build:

import datarobot as dr

build_info = dr.CustomModelVersionDependencyBuild.start_build(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
    max_wait=3600,  # 1 hour timeout
)

build_info.build_status
>>> 'success'

To start a custom model version dependency build without blocking a program until the test finishes, set max_wait to None:

import datarobot as dr

build_info = dr.CustomModelVersionDependencyBuild.start_build(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
    max_wait=None,
)

build_info.build_status
>>> 'submitted'

# after some time
build_info.refresh()
build_info.build_status
>>> 'success'

In case the build fails, or you are just curious, do the following to retrieve the build log once complete:

print(build_info.get_log())

To cancel a custom model version dependency build, simply run:

build_info.cancel()

Manage custom model tests

A custom model test represents testing performed on custom models.

Create a custom model test

To create a custom model test:

import datarobot as dr

path_to_dataset = '/home/user/Documents/testDataset.csv'
dataset = dr.Dataset.create_from_file(file_path=path_to_dataset)

custom_model_test = dr.CustomModelTest.create(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
    dataset_id=dataset.id,
    max_wait=3600,  # 1 hour timeout
)

custom_model_test.overall_status
>>> 'succeeded'

or, with k8s resources:

import datarobot as dr

path_to_dataset = '/home/user/Documents/testDataset.csv'
dataset = dr.Dataset.create_from_file(file_path=path_to_dataset)

custom_model_test = dr.CustomModelTest.create(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
    dataset_id=dataset.id,
    max_wait=3600,  # 1 hour timeout
    maximum_memory=1024*1024*1024,
)

custom_model_test.overall_status
>>> 'succeeded'

To start a custom model test without blocking a program until the test finishes, set max_wait to None:

import datarobot as dr

path_to_dataset = '/home/user/Documents/testDataset.csv'
dataset = dr.Dataset.create_from_file(file_path=path_to_dataset)

custom_model_test = dr.CustomModelTest.create(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
    dataset_id=dataset.id,
    max_wait=None,
)

custom_model_test.overall_status
>>> 'in_progress'

# after some time
custom_model_test.refresh()
custom_model_test.overall_status
>>> 'succeeded'

Running a custom model test uses the custom model version’s base image with its dependencies installed as an execution environment. To start a custom model test using an execution environment as-is, without the model’s dependencies installed, supply an environment ID and (optionally) an environment version ID:

import datarobot as dr

path_to_dataset = '/home/user/Documents/testDataset.csv'
dataset = dr.Dataset.create_from_file(file_path=path_to_dataset)

custom_model_test = dr.CustomModelTest.create(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
    dataset_id=dataset.id,
    max_wait=3600,  # 1 hour timeout
)

custom_model_test.overall_status
>>> 'succeeded'

In case a test fails, do the following to examine details of the failure:

for name, test in custom_model_test.detailed_status.items():
    print('Test: {}'.format(name))
    print('Status: {}'.format(test['status']))
    print('Message: {}'.format(test['message']))

print(custom_model_test.get_log())

To cancel a custom model test:

custom_model_test.cancel()

To start a custom model test for an unstructured custom model, dataset details should not be provided:

import datarobot as dr

custom_model_test = dr.CustomModelTest.create(
    custom_model_id=custom_model.id,
    custom_model_version_id=model_version.id,
)

List custom model tests

To list the custom model tests available to the user:

import datarobot as dr

dr.CustomModelTest.list(custom_model_id=custom_model.id)
>>> [CustomModelTest('5ec262604024031bed5aaa16')]

Retrieve a custom model test

To retrieve a specific custom model test:

import datarobot as dr

dr.CustomModelTest.get(custom_model_test_id='5ec262604024031bed5aaa16')
>>> CustomModelTest('5ec262604024031bed5aaa16')