LLM Generation

class datarobot.models.genai.custom_model_validation.CustomModelValidation

The validation record checking the ability of the deployment to serve as a custom model LLM, custom model vector database, or custom model embedding.

Variables:

  • id (str) – The ID of the validation.

  • prompt_column_name (str) – The name of the column the deployed model uses for prompt text input.

  • target_column_name (str) – The name of the column the deployed model uses for prediction output.

  • deployment_id (str) – The ID of the deployment.

  • model_id (str) – The ID of the underlying deployed model, which can be found using Deployment.model[“id”].

  • validation_status (str) – Can be TESTING, FAILED, or PASSED. Only PASSED is allowed for use.

  • deployment_access_data (dict, optional) – The data that will be used for accessing the deployment prediction server. This field is only available for deployments that pass validation. Dict fields are as follows: - prediction_api_url - The URL for the deployment prediction server. - datarobot_key - The first of two auth headers for the prediction server. - authorization_header - The second of two auth headers for the prediction server. - input_type - The input type the model expects, either JSON or CSV. - model_type - The target type of the deployed custom model.

  • tenant_id (str) – The creating user’s tenant ID.

  • name (str) – The display name of the validated custom model.

  • creation_date (str) – The creation date of the validation (ISO 8601 formatted).

  • user_id (str) – The ID of the creating user.

  • error_message (Optional[str]) – Additional information for the errored validation.

  • deployment_name (Optional[str]) – The name of the validated deployment.

  • user_name (Optional[str]) – The name of the creating user.

  • use_case_id (Optional[str]) – The ID of the Use Case associated with the validation.

  • prediction_timeout (int) – The timeout, in seconds, for the prediction API used in this custom model validation.

classmethod get(validation_id)

Get the validation record by id.

Parameters:

validation_id (Union[CustomModelValidation, str]) – The CustomModelValidation to retrieve, either CustomModelValidation or validation ID.

Return type:

CustomModelValidation

classmethod get_by_values(prompt_column_name, target_column_name, deployment_id, model_id)

Get the validation record by field values.

Parameters:

  • prompt_column_name (str) – The name of the column the deployed model uses for prompt text input.

  • target_column_name (str) – The name of the column the deployed model uses for prediction output.

  • deployment_id (str) – The ID of the deployment.

  • model_id (str) – The ID of the underlying deployed model.

Return type:

CustomModelValidation

classmethod list(prompt_column_name=None, target_column_name=None, deployment=None, model=None, use_cases=None, playground=None, completed_only=False, search=None, sort=None)

List the validation records by field values.

Parameters:

  • prompt_column_name (Optional[str], optional) – The column name the deployed model expects as the input.

  • target_column_name (Optional[str], optional) – The target name that the deployed model will output.

  • deployment (Optional[Union[Deployment, str]], optional) – The returned validations are filtered to those associated with a specific deployment if specified, either Deployment or the deployment ID.

  • model_id (Optional[Union[Model, str]], optional) – The returned validations are filtered to those associated with a specific model if specified, either Model or model ID.

  • use_cases (Optional[list[Union[UseCase, str]]], optional) – The returned validations are filtered to those associated with specific Use Cases if specified, either UseCase objects or the Use Case IDs.

  • playground_id (Optional[Union[Playground, str]], optional) – The returned validations are filtered to those used in a specific playground if specified, either Playground or playground ID.

  • completed_only (Optional[bool]) – Whether to retrieve only completed validations.

  • search (Optional[str], optional) – String for filtering validations. Validations that contain the string in name will be returned.

  • sort (Optional[str], optional) – Property to sort validations by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-name’. Currently supported options are listed in ListCustomModelValidationsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in validations being returned in order of creation time descending.

Return type:

List[CustomModelValidation]

classmethod revalidate(validation_id)

Revalidate an unlinked custom model vector database or LLM. This method is helpful when a deployment used as vector database or LLM is accidentally replaced with another model that stopped complying with the response schema requirements.

Replace the deployment’s model with a complying model and call this method instead of creating a new custom model validation from scratch.

Another application is if the API token used to create a validation record got revoked and no longer can be used to call the deployment. Calling revalidate will update the validation record with the token currently in use.

Parameters:

validation_id (str) – The ID of the CustomModelValidation for revalidation.

Return type:

CustomModelValidation

delete()

Delete the custom model validation.

Return type:

None

class datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation

Validation record checking the ability of the deployment to serve as a custom model LLM.

Variables:

  • id (str) – The ID of the validation.

  • prompt_column_name (str) – The name of the column the deployed model uses for prompt text input.

  • target_column_name (str) – The name of the column the deployed model uses for prediction output.

  • chat_model_id (Optional[str]) – The model ID to specify when calling the chat completion API of the deployment.

  • deployment_id (str) – The ID of the deployment.

  • model_id (str) – The ID of the underlying deployed model, which can be found using Deployment.model[“id”].

  • validation_status (str) – Can be TESTING, FAILED, or PASSED. Only PASSED is allowed for use.

  • deployment_access_data (dict, optional) – The data that will be used for accessing the deployment prediction server. This field is only available for deployments that pass validation. Dict fields are as follows: - prediction_api_url - The URL for the deployment prediction server. - datarobot_key - The first of two auth headers for the prediction server. - authorization_header - The second of two auth headers for the prediction server. - input_type - The input type the model expects, either JSON or CSV. - model_type - The target type of the deployed custom model.

  • tenant_id (str) – The creating user’s tenant ID.

  • name (str) – The display name of the validated custom model.

  • creation_date (str) – The creation date of the validation (ISO 8601 formatted).

  • user_id (str) – The ID of the creating user.

  • error_message (Optional[str]) – Additional information for the errored validation.

  • deployment_name (Optional[str]) – The name of the validated deployment.

  • user_name (Optional[str]) – The name of the creating user.

  • use_case_id (Optional[str]) – The ID of the Use Case associated with the validation.

  • prediction_timeout (int) – The timeout in seconds for the prediction API used in this custom model validation.

classmethod create(deployment_id, model=None, use_case=None, name=None, wait_for_completion=False, prediction_timeout=None, prompt_column_name=None, target_column_name=None, chat_model_id=None)

Start the validation of the deployment that will serve as an LLM.

Parameters:

  • deployment_id (Union[Deployment, str]) – The deployment to validate, either Deployment or the deployment ID.

  • model (Optional[Union[Model, str]], optional) – The specific model within the deployment, either Model or the model ID. If not specified, the underlying model ID will be automatically derived from the deployment information.

  • use_case (Optional[Union[UseCase, str]], optional) – The Use Case to link the validation to, either UseCase or the Use Case ID.

  • name (Optional[str], optional) – The name of the validation.

  • wait_for_completion (bool) – If set to True, the code will wait for the validation job to complete before returning results. If the job does not finish in 10 minutes, this method call raises a timeout error. If set to False, the code does not wait for the job to complete. Instead, CustomModelLLMValidation.get can be used to poll for the status of the job using the validation ID returned by the method.

  • prediction_timeout (Optional[int], optional) – The timeout, in seconds, for the prediction API used in this custom model validation.

  • prompt_column_name (Optional[str], optional) – The name of the column the deployed model uses for prompt text input. This value is used to call the Prediction API of the deployment. For LLM deployments that support the chat completion API, it is recommended to specify chatModelId instead.

  • target_column_name (Optional[str], optional) – The name of the column the deployed model uses for prediction output. This value is used to call the Prediction API of the deployment. For LLM deployments that support the chat completion API, it is recommended to specify chatModelId instead.

  • chat_model_id (Optional[str], optional) – The model ID to specify when calling the chat completion API of the deployment. If this parameter is specified, the deployment must support the chat completion API.

Return type:

CustomModelLLMValidation

update(name=None, prompt_column_name=None, target_column_name=None, chat_model_id=None, deployment=None, model=None, prediction_timeout=None)

Update a custom model validation.

Parameters:

  • name (Optional[str], optional) – The new name of the custom model validation.

  • prompt_column_name (Optional[str], optional) – The new name of the prompt column.

  • target_column_name (Optional[str], optional) – The new name of the target column.

  • chat_model_id (Optional[str], optional) – The new model ID to specify when calling the chat completion API of the deployment.

  • deployment (Optional[Union[Deployment, str]]) – The new deployment to validate.

  • model (Optional[Union[Model, str]], optional) – The new model within the deployment to validate.

  • prediction_timeout (Optional[int], optional) – The new timeout, in seconds, for the prediction API used in this custom model validation.

Return type:

CustomModelLLMValidation

class datarobot.models.genai.llm_blueprint.LLMBlueprint

Metadata for a DataRobot GenAI LLM blueprint.

Variables:

  • id (str) – The LLM blueprint ID.

  • name (str) – The LLM blueprint name.

  • description (str) – A description of the LLM blueprint.

  • is_saved (bool) – Whether the LLM blueprint is saved (meaning the settings are locked and blueprint is eligible for use with ComparisonPrompts).

  • is_starred (bool) – Whether the LLM blueprint is starred.

  • playground_id (str) – The ID of the playground associated with the LLM blueprint.

  • llm_id (str or None) – The ID of the LLM type. If not None, this must be one of the IDs returned by LLMDefinition.list for this user.

  • llm_name (str or None) – The name of the LLM.

  • llm_settings (dict or None) – The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the external model LLM validation. - external_llm_context_size - The external LLM’s context size, in tokens, for external model LLM blueprints.

  • creation_date (str) – The date the playground was created.

  • creation_user_id (str) – The ID of the user creating the playground.

  • creation_user_name (str) – The name of the user creating the playground.

  • last_update_date (str) – The date the playground was last updated.

  • last_update_user_id (str) – The ID of the user who most recently updated the playground.

  • prompt_type (PromptType) – The prompting strategy for the LLM blueprint. Currently supported options are listed in PromptType.

  • vector_database_id (str or None) – The ID of the vector database, if any, associated with the LLM blueprint.

  • vector_database_settings (VectorDatabaseSettings or None) – The settings for the vector database, if any, associated with the LLM blueprint.

  • vector_database_name (str or None) – The name of the vector database associated with the LLM blueprint, if any.

  • vector_database_status (str or None) – The status of the vector database, if any, associated with the LLM blueprint.

  • vector_database_error_message (str or None) – The error message for the vector database, if any, associated with the LLM blueprint.

  • vector_database_error_resolution (str or None) – The resolution for the vector database error, if any, associated with the LLM blueprint.

  • custom_model_llm_validation_status (str or None) – The status of the custom model LLM validation if the llm_id is ‘custom-model’.

  • custom_model_llm_error_message (str or None) – The error message for the custom model LLM, if any.

  • custom_model_llm_error_resolution (str or None) – The resolution for the custom model LLM error, if any.

classmethod create(playground, name, prompt_type=PromptType.CHAT_HISTORY_AWARE, description='', llm=None, llm_settings=None, vector_database=None, vector_database_settings=None)

Create a new LLM blueprint.

Parameters:

  • playground (Playground or str) – The playground associated with the created LLM blueprint. Accepts playground or playground ID.

  • name (str) – The LLM blueprint name.

  • prompt_type (PromptType, optional) – Prompting type of the LLM blueprint, by default PromptType.CHAT_HISTORY_AWARE.

  • description (Optional[str]) – An optional description for the LLM blueprint, otherwise null.

  • llm (LLMDefinition, str, or None, optional) – The LLM to use for the blueprint, either LLMDefinition or LLM ID.

  • llm_settings (dict or None) – The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.

  • vector_database (VectorDatabase, str, or None, optional) – The vector database to use with this LLM blueprint, either VectorDatabase or vector database ID.

  • vector_database_settings (VectorDatabaseSettings or None, optional) – The settings for the vector database, if any.

Returns:

llm_blueprint – The created LLM blueprint.

Return type:

LLMBlueprint

classmethod create_from_llm_blueprint(llm_blueprint, name, description='')

Create a new LLM blueprint from an existing LLM blueprint.

Parameters:

  • llm_blueprint (LLMBlueprint or str) – The LLM blueprint to use to create the new LLM blueprint. Accepts LLM blueprint or LLM blueprint ID.

  • name (str) – LLM blueprint name.

  • description (Optional[str]) – Description of the LLM blueprint, by default “”.

Returns:

llm_blueprint – The created LLM blueprint.

Return type:

LLMBlueprint

classmethod get(llm_blueprint_id)

Retrieve a single LLM blueprint.

Parameters:

llm_blueprint_id (str) – The ID of the LLM blueprint you want to retrieve.

Returns:

llm_blueprint – The requested LLM blueprint.

Return type:

LLMBlueprint

classmethod list(playground=None, llms=None, vector_databases=None, is_saved=None, is_starred=None, sort=None)

Lists all LLM blueprints available to the user. If the playground is specified, then the results are restricted to the LLM blueprints associated with the playground. If the LLMs are specified, then the results are restricted to the LLM blueprints using those LLM types. If vector_databases are specified, then the results are restricted to the LLM blueprints using those vector databases.

Parameters:

  • playground (Optional[Union[Playground, str]], optional) – The returned LLM blueprints are filtered to those associated with a specific playground if it is specified. Accepts either the entity or the ID.

  • llms (Optional[list[Union[LLMDefinition, str]]], optional) – The returned LLM blueprints are filtered to those associated with the LLM types specified. Accepts either the entity or the ID.

  • vector_databases (Optional[list[Union[VectorDatabase, str]]], optional) – The returned LLM blueprints are filtered to those associated with the vector databases specified. Accepts either the entity or the ID.

  • is_saved (Optional[bool], optional) – The returned LLM blueprints are filtered to those matching is_saved.

  • is_starred (Optional[bool], optional) – The returned LLM blueprints are filtered to those matching is_starred.

  • sort (Optional[str]) – Property to sort LLM blueprints by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListLLMBlueprintsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in LLM blueprints being returned in order of creation time descending.

Returns:

playgrounds – A list of playgrounds available to the user.

Return type:

list[Playground]

update(name=None, description=None, llm=None, llm_settings=None, vector_database=None, vector_database_settings=None, is_saved=None, is_starred=None, prompt_type=None, remove_vector_database=False)

Update the LLM blueprint.

Parameters:

  • name (str or None, optional) – The new name for the LLM blueprint.

  • description (str or None, optional) – The new description for the LLM blueprint.

  • llm (Optional[Union[LLMDefinition, str]], optional) – The new LLM type for the LLM blueprint.

  • llm_settings (Optional[dict], optional) – The new LLM settings for the LLM blueprint. These must match the LLMSettings returned from the LLMDefinition.list method for the LLM type used for this LLM blueprint but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.

  • vector_database (Optional[Union[VectorDatabase, str]], optional) – The new vector database for the LLM blueprint.

  • vector_database_settings (Optional[VectorDatabaseSettings], optional) – The new vector database settings for the LLM blueprint.

  • is_saved (Optional[bool], optional) – The new is_saved attribute for the LLM blueprint (meaning the settings are locked and blueprint is eligible for use with ComparisonPrompts).

  • is_starred (Optional[bool], optional) – The new setting for whether the LLM blueprint is starred.

  • prompt_type (PromptType, optional) – The new prompting type of the LLM blueprint.

  • remove_vector_database (Optional[bool], optional) – Whether to remove the vector database from the LLM blueprint.

Returns:

llm_blueprint – The updated LLM blueprint.

Return type:

LLMBlueprint

delete()

Delete the single LLM blueprint.

Return type:

None

register_custom_model(prompt_column_name=None, target_column_name=None, llm_test_configuration_ids=None)

Create a new CustomModelVersion. This registers a custom model from the LLM blueprint.

Parameters:

  • prompt_column_name (Optional[str]) – The column name of the prompt text.

  • target_column_name (Optional[str]) – The column name of the response text.

  • llm_test_configuration_ids (List[str], optional) – The IDs of the LLM test configurations to execute.

Returns:

custom_model – The registered custom model.

Return type:

CustomModelVersion

class datarobot.models.genai.llm.LLMDefinition

Metadata for a DataRobot GenAI LLM.

Variables:

  • id (str) – Language model type ID.

  • name (str) – Language model name.

  • description (str) – Description of the language model.

  • vendor (str) – Name of the vendor for this model.

  • license (str) – License for this model.

  • supported_languages (str) – Languages supported by this model.

  • settings (list of LLMSettingDefinition) – Settings for this model

  • context_size (int) – The context size for this model

  • retirement_date (datetime.date, optional) – When the LLM is expected to be retired and no longer available for submitting new prompts.

  • is_deprecated (bool, optional) – Whether the LLM is deprecated and will be removed when it is past the retirement date.

  • is_active (bool, optional) – Whether the LLM is active.

classmethod list(use_case=None, as_dict=True)

List all large language models (LLMs) available to the user.

Parameters:

use_case (Optional[UseCase or str], optional) – The returned LLMs, including external LLMs, available for the specified Use Case. Accepts either the entity or the Use CaseID.

Returns:

llms – A list of large language models (LLMs) available to the user.

Return type:

list[LLMDefinition] or list[LLMDefinitionDict]

class datarobot.models.genai.llm.LLMDefinitionDict

Dict representation of LLMDefinition.