Generative AI

class datarobot.models.genai.chat.Chat

Metadata for a DataRobot GenAI chat.

Variables:

id (str) – The chat ID.
name (str) – The chat name.
llm_blueprint_id (str) – The ID of the LLM blueprint associated with the chat.
is_frozen (bool) – Checks whether the chat is frozen. Prompts cannot be submitted to frozen chats.
creation_date (str) – The date when the chat was created.
creation_user_id (str) – The ID of the creating user.
warning (str or None, optional) – The warning about the contents of the chat.
prompts_count (int) – The number of chat prompts in the chat.

classmethod create(name, llm_blueprint)

Creates a new chat.

Parameters:

name (str) – The chat name.
llm_blueprint (LLMBlueprint or str) – The LLM blueprint associated with the created chat, either LLM blueprint or ID.

Returns:

chat – The created chat.

Return type:

Chat

classmethod get(chat)

Retrieve a single chat.

Parameters:: chat (Chat or str) – The chat you want to retrieve. Accepts chat or chat ID.
Returns:: chat – The requested chat.
Return type:: Chat

classmethod list(llm_blueprint=None, sort=None)

List all chats available to the user. If the LLM blueprint is specified, results are restricted to only those chats associated with the LLM blueprint.

Parameters:

llm_blueprint (Optional[Union[LLMBlueprint, str]], optional) – Returns only those chats associated with a particular LLM blueprint, specified by either the entity or the ID.
sort (Optional[str]) – The property to sort chats by. Prefix the attribute name with a dash ( - ) to sort responses in descending order, (for example, ‘-name’). Supported options are listed in ListChatsSortQueryParams, but the values can differ depending on platform version. The default sort parameter is None, which results in chats returning in order of creation time, descending.

Returns:

chats – Returns a list of chats.

Return type:

list[Chat]

delete()

Delete the single chat.

Return type:: None

update(name)

Update the chat.

Parameters:: name (str) – The new name for the chat.
Returns:: chat – The updated chat.
Return type:: Chat

class datarobot.models.genai.chat_prompt.ChatPrompt

Metadata for a DataRobot GenAI chat prompt.

Variables:

id (str) – Chat prompt ID.
text (str) – The prompt text.
llm_blueprint_id (str) – ID of the LLM blueprint associated with the chat prompt.
llm_id (str) – ID of the LLM type. This must be one of the IDs returned by LLMDefinition.list for this user.
llm_settings (dict or None) – The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list, but this typically has dict fields. Either: - system_prompt - The system prompt that influences the LLM responses. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Sets whether the model considers next tokens with top_p probability mass. or - system_prompt - The system prompt that influences the LLM responses. - validation_id - The ID of the external model LLM validation. - external_llm_context_size - The external LLM’s context size, in tokens, for external model-based LLM blueprints.
creation_date (str) – The date the chat prompt was created.
creation_user_id (str) – ID of the creating user.
vector_database_id (str or None) – ID of the vector database associated with the LLM blueprint, if any.
vector_database_settings (VectorDatabaseSettings or None) – The settings for the vector database associated with the LLM blueprint, if any.
result_metadata (ResultMetadata or None) – Metadata for the result of the chat prompt submission.
result_text (str or None) – The result text from the chat prompt submission.
confidence_scores (ConfidenceScores or None) – The confidence scores if there is a vector database associated with the chat prompt.
citations (list[Citation]) – List of citations from text retrieved from the vector database, if any.
execution_status (str) – The execution status of the chat prompt.
chat_id (Optional[str]) – ID of the chat associated with the chat prompt.
chat_context_id (Optional[str]) – The ID of the chat context for the chat prompt.
chat_prompt_ids_included_in_history (Optional[list[str]]) – The IDs of the chat prompts included in the chat history for this chat prompt.
metadata_filter (Optional[Dict[str, Any] | None]) –
The metadata filter to apply to the vector database.

Supports: - None or empty dict (no filters): Considers all documents - Multiple field filters (implicit AND): {“a”: 1, “b”: “b”} - Comparison operators: {“field”: {“$gt”: 5}} - Logical operators: {“$and”: […], “$or”: […]} - Nested combinations of the above

Comparison operators: - $eq: equal to (string, int, float, bool) - $ne: not equal to (string, int, float, bool) - $gt: greater than (int, float) - $gte: greater than or equal to (int, float) - $lt: less than (int, float) - $lte: less than or equal to (int, float) - $in: a value is in list (string, int, float, bool) - $nin: a value is not in list (string, int, float, bool) - $contains: a string contains a value (string) - $not_contains: a string does not contain a value (string)

classmethod create(text, llm_blueprint=None, chat=None, llm=None, llm_settings=None, vector_database=None, vector_database_settings=None, wait_for_completion=False, metadata_filter=None)

Create a new ChatPrompt. This submits the prompt text to the LLM. Either llm_blueprint or chat is required.

Parameters:

text (str) – The prompt text.
llm_blueprint (LLMBlueprint or str or None, optional) – The LLM blueprint associated with the created chat prompt, either LLMBlueprint or LLM blueprint ID.
chat (Chat or str or None, optional) – The chat associated with the created chat prompt, either Chat or chat ID.
llm (LLMDefinition, str, or None, optional) – The LLM to use for the chat prompt, either LLMDefinition or LLM blueprint ID.
llm_settings (dict or None) – LLM settings to use for the chat prompt. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.
vector_database (VectorDatabase, str, or None, optional) – The vector database to use with this chat prompt submission, either VectorDatabase or vector database ID.
vector_database_settings (VectorDatabaseSettings or None, optional) – Settings for the vector database, if any.
wait_for_completion (bool) – If set to True, chat prompt result response limit is up to 10 minutes, raising a timeout error after that. Otherwise, check current status by using ChatPrompt.get with returned ID.
metadata_filter (Optional[Dict[str, Any] | None]) –
The metadata filter to apply to the vector database.

Supports: - None or empty dict (no filters): Considers all documents - Multiple field filters (implicit AND): {“a”: 1, “b”: “b”} - Comparison operators: {“field”: {“$gt”: 5}} - Logical operators: {“$and”: […], “$or”: […]} - Nested combinations of the above

Comparison operators: - $eq: equal to (string, int, float, bool) - $ne: not equal to (string, int, float, bool) - $gt: greater than (int, float) - $gte: greater than or equal to (int, float) - $lt: less than (int, float) - $lte: less than or equal to (int, float) - $in: a value is in list (string, int, float, bool) - $nin: a value is not in list (string, int, float, bool) - $contains: a string contains a value (string) - $not_contains: a string does not contain a value (string)

Returns:

chat_prompt – The created chat prompt.

Return type:

ChatPrompt

update(custom_metrics=None, feedback_metadata=None)

Update the chat prompt.

Parameters:

custom_metrics (Optional[list[MetricMetadata]], optional) – The new custom metrics to add to the chat prompt.
feedback_metadata (Optional[FeedbackMetadata], optional) – The new feedback to add to the chat prompt.

Returns:

chat_prompt – The updated chat prompt.

Return type:

ChatPrompt

classmethod get(chat_prompt)

Retrieve a single chat prompt.

Parameters:: chat_prompt (ChatPrompt or str) – The chat prompt you want to retrieve, either ChatPrompt or chat prompt ID.
Returns:: chat_prompt – The requested chat prompt.
Return type:: ChatPrompt

classmethod list(llm_blueprint=None, playground=None, chat=None)

List all chat prompts available to the user. If the llm_blueprint, playground, or chat is specified then the results are restricted to the chat prompts associated with that entity.

Parameters:

llm_blueprint (Optional[Union[LLMBlueprint, str]], optional) – The returned chat prompts are filtered to those associated with a specific LLM blueprint if it is specified. Accepts either LLMBlueprint or LLM blueprint ID.
playground (Optional[Union[Playground, str]], optional) – The returned chat prompts are filtered to those associated with a specific playground if it is specified. Accepts either Playground or playground ID.
chat (Optional[Union[Chat, str]], optional) – The returned chat prompts are filtered to those associated with a specific chat if it is specified. Accepts either Chat or chat ID.

Returns:

chat_prompts – A list of chat prompts available to the user.

Return type:

list[ChatPrompt]

delete()

Delete the single chat prompt.

Return type:: None

create_llm_blueprint(name, description='')

Create a new LLM blueprint from an existing chat prompt.

Parameters:

name (str) – LLM blueprint name.
description (Optional[str]) – Description of the LLM blueprint, by default “”.

Returns:

llm_blueprint – The created LLM blueprint.

Return type:

LLMBlueprint

class datarobot.models.genai.comparison_chat.ComparisonChat

Metadata for a DataRobot GenAI comparison chat.

Variables:

id (str) – The comparison chat ID.
name (str) – The comparison chat name.
playground_id (str) – The ID of the playground associated with the comparison chat.
creation_date (str) – The date when the comparison chat was created.
creation_user_id (str) – The ID of the creating user.

classmethod create(name, playground)

Creates a new comparison chat.

Parameters:

name (str) – The comparison chat name.
playground (Playground or str) – The playground associated with the created comparison chat, either Playground or playground ID.

Returns:

comparison_chat – The created comparison chat.

Return type:

ComparisonChat

classmethod get(comparison_chat)

Retrieve a single comparison chat.

Parameters:: comparison_chat (ComparisonChat or str) – The comparison chat you want to retrieve. Accepts ComparisonChat or comparison chat ID.
Returns:: comparison_chat – The requested comparison chat.
Return type:: ComparisonChat

classmethod list(playground=None, sort=None)

List all comparison chats available to the user. If the playground is specified, results are restricted to only those comparison chats associated with the playground.

Parameters:

playground (Optional[Union[Playground, str]], optional) – Returns only those comparison chats associated with a particular playground, specified by either the Playground or the playground ID.
sort (Optional[str]) – The property to sort comparison chats by. Prefix the attribute name with a dash ( - ) to sort responses in descending order, (for example, ‘-name’). Supported options are listed in ListComparisonChatsSortQueryParams, but the values can differ depending on platform version. The default sort parameter is None, which results in comparison chats returning in order of creation time, descending.

Returns:

comparison_chats – Returns a list of comparison chats.

Return type:

list[ComparisonChat]

delete()

Delete the single comparison chat.

Return type:: None

update(name)

Update the comparison chat.

Parameters:: name (str) – The new name for the comparison chat.
Returns:: comparison_chat – The updated comparison chat.
Return type:: ComparisonChat

class datarobot.models.genai.comparison_prompt.ComparisonPrompt

Metadata for a DataRobot GenAI comparison prompt.

Variables:

id (str) – Comparison prompt ID.
text (str) – The prompt text.
results (list[ComparisonPromptResult]) – The list of results for individual LLM blueprints that are part of the comparison prompt.
creation_date (str) – The date when the playground was created.
creation_user_id (str) – ID of the creating user.
comparison_chat_id (str) – The ID of the comparison chat this comparison prompt is associated with.
metadata_filter (Optional[Dict[str, Any] | None]) –
The metadata filter to apply to the vector database.

Supports: - None or empty dict (no filters): Considers all documents - Multiple field filters (implicit AND): {“a”: 1, “b”: “b”} - Comparison operators: {“field”: {“$gt”: 5}} - Logical operators: {“$and”: […], “$or”: […]} - Nested combinations of the above

Comparison operators: - $eq: equal to (string, int, float, bool) - $ne: not equal to (string, int, float, bool) - $gt: greater than (int, float) - $gte: greater than or equal to (int, float) - $lt: less than (int, float) - $lte: less than or equal to (int, float) - $in: a value is in list (string, int, float, bool) - $nin: a value is not in list (string, int, float, bool) - $contains: a string contains a value (string) - $not_contains: a string does not contain a value (string)

update(additional_llm_blueprints=None, wait_for_completion=False, feedback_result=None, **kwargs)

Update the comparison prompt.

Parameters:: additional_llm_blueprints (list[LLMBlueprint or str]) – The additional LLM blueprints you want to submit the comparison prompt.
Returns:: comparison_prompt – The updated comparison prompt.
Return type:: ComparisonPrompt

classmethod create(llm_blueprints, text, comparison_chat=None, wait_for_completion=False, metadata_filter=None)

Create a new ComparisonPrompt. This submits the prompt text to the LLM blueprints that are specified.

Parameters:

llm_blueprints (list[LLMBlueprint or str]) – The LLM blueprints associated with the created comparison prompt. Accepts LLM blueprints or IDs.
text (str) – The prompt text.
comparison_chat (Optional[ComparisonChat or str], optional) – The comparison chat to add the comparison prompt to. Accepts ComparisonChat or comparison chat ID.
wait_for_completion (bool) – If set to True code will wait for the chat prompt job to complete before returning the result (up to 10 minutes, raising timeout error after that). Otherwise, you can check current status by using ChatPrompt.get with returned ID.
metadata_filter (Optional[Dict[str, Any] | None]) –
The metadata filter to apply to the vector database.

Supports: - None or empty dict (no filters): Considers all documents - Multiple field filters (implicit AND): {“a”: 1, “b”: “b”} - Comparison operators: {“field”: {“$gt”: 5}} - Logical operators: {“$and”: […], “$or”: […]} - Nested combinations of the above

Comparison operators: - $eq: equal to (string, int, float, bool) - $ne: not equal to (string, int, float, bool) - $gt: greater than (int, float) - $gte: greater than or equal to (int, float) - $lt: less than (int, float) - $lte: less than or equal to (int, float) - $in: a value is in list (string, int, float, bool) - $nin: a value is not in list (string, int, float, bool) - $contains: a string contains a value (string) - $not_contains: a string does not contain a value (string)

Returns:

comparison_prompt – The created comparison prompt.

Return type:

ComparisonPrompt

classmethod get(comparison_prompt)

Retrieve a single comparison prompt.

Parameters:: comparison_prompt (str) – The comparison prompt you want to retrieve. Accepts entity or ID.
Returns:: comparison_prompt – The requested comparison prompt.
Return type:: ComparisonPrompt

classmethod list(llm_blueprints=None, comparison_chat=None)

List all comparison prompts available to the user that include the specified LLM blueprints or from the specified comparison chat.

Parameters:

llm_blueprints (Optional[List[Union[LLMBlueprint, str]]], optional) – The returned comparison prompts are only those associated with the specified LLM blueprints. Accepts either LLMBlueprint or LLM blueprint ID.
comparison_chat (Optional[Union[ComparisonChat, str]], optional) – The returned comparison prompts are only those associated with the specified comparison chat. Accepts either ComparisonChat or comparison chat ID.

Returns:

comparison_prompts – A list of comparison prompts available to the user that use the specified LLM blueprints.

Return type:

list[ComparisonPrompt]

delete()

Delete the single comparison prompt.

Return type:: None

class datarobot.models.genai.custom_model_validation.CustomModelValidation

Validation record checking the ability of the deployment to serve as a custom model LLM or vector database.

Variables:

prompt_column_name (str) – The column name that the deployed model expects as the input.
target_column_name (str) – The target name that the deployed model will output.
deployment_id (str) – ID of the deployment.
model_id (str) – ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].
validation_status (str) – Can be TESTING, FAILED, or PASSED. Only PASSED is allowed for use.
deployment_access_data (dict, optional) – Data that will be used for accessing deployment prediction server. Only available for deployments that pass validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - First of two auth headers for the prediction server. - authorization_header - Second of two auth headers for the prediction server. - input_type - Either JSON or CSV - input type model expects. - model_type - Target type of deployed custom model.
tenant_id (str) – Creating user’s tenant ID.
error_message (Optional[str]) – Additional information for errored validation.
deployment_name (Optional[str]) – The name of the deployment that is validated.
user_name (Optional[str]) – The name of the user
use_case_id (Optional[str]) – The ID of the Use Case associated with the validation.
prediction_timeout (int) – The timeout, in seconds, for the prediction API used in this custom model validation.

classmethod get(validation_id)

Get the validation record by id.

Parameters:: validation_id (Union[CustomModelValidation, str]) – The CustomModelValidation to retrieve, either CustomModelValidation or validation ID.
Return type:: CustomModelValidation

classmethod get_by_values(prompt_column_name, target_column_name, deployment_id, model_id)

Get the validation record by field values.

Parameters:

prompt_column_name (str) – The column name the deployed model expect as the input.
target_column_name (str) – The target name deployed model will output.
deployment_id (str) – ID of the deployment.
model_id (str) – ID of the underlying deployment model.

Return type:

CustomModelValidation

classmethod list(prompt_column_name=None, target_column_name=None, deployment=None, model=None, use_cases=None, playground=None, completed_only=False, search=None, sort=None)

List the validation records by field values.

Parameters:

prompt_column_name (Optional[str], optional) – The column name the deployed model expects as the input.
target_column_name (Optional[str], optional) – The target name that the deployed model will output.
deployment (Optional[Union[Deployment, str]], optional) – The returned validations are filtered to those associated with a specific deployment if specified, either Deployment or deployment ID.
model_id (Optional[Union[Model, str]], optional) – The returned validations are filtered to those associated with a specific model if specified, either Model or model ID.
use_cases (Optional[list[Union[UseCase, str]]], optional) – The returned validations are filtered to those associated with specific Use Cases if specified, either UseCase or Use Case IDs.
playground_id (Optional[Union[Playground, str]], optional) – The returned validations are filtered to those used in a specific playground if specified, either Playground or playground ID.
completed_only (Optional[bool]) – Whether to retrieve only completed validations.
search (Optional[str], optional) – String for filtering validations. Validations that contain the string in name will be returned.
sort (Optional[str], optional) – Property to sort validations by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-name’. Currently supported options are listed in ListCustomModelValidationsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in validations being returned in order of creation time descending.

Return type:

List[CustomModelValidation]

classmethod create(prompt_column_name, target_column_name, deployment_id, model=None, use_case=None, name=None, wait_for_completion=False, prediction_timeout=None)

Start the validation of deployment to serve as a vector database or LLM.

Parameters:

prompt_column_name (str) – The column name the deployed model expect as the input.
target_column_name (str) – The target name that the deployed model will output.
deployment_id (Union[Deployment, str]) – The deployment to validate, either Deployment or deployment ID.
model (Optional[Union[Model, str]], optional) – The specific model within the deployment, either Model or model ID. If not specified, the underlying model ID will be derived from the deployment info automatically.
use_case (Optional[Union[UseCase, str]], optional) – The Use Case to link the validation to, either UseCase or Use Case ID.
name (Optional[str], optional) – The name of the validation.
wait_for_completion (bool) – If set to True code will wait for the validation job to complete before returning the result (up to 10 minutes, raising timeout error after that). Otherwise, you can check current validation status by using CustomModelValidation.get with returned ID.
prediction_timeout (Optional[int], optional) – The timeout, in seconds, for the prediction API used in this custom model validation.

Return type:

CustomModelValidation

classmethod revalidate(validation_id)

Revalidate an unlinked custom model vector database or LLM. This method is useful when a deployment used as vector database or LLM is accidentally replaced with another model that stopped complying with the vector database or LLM requirements. Replace the model back and call this method instead of creating a new custom model validation from scratch. Another use case for this is when the API token used to create a validation record got revoked and no longer can be used by vector database / LLM to call custom model deployment. Calling revalidate will update the validation record with the token currently in use.

Parameters:: validation_id (str) – The ID of the CustomModelValidation for revalidation.
Return type:: CustomModelValidation

update(name=None, prompt_column_name=None, target_column_name=None, deployment=None, model=None, prediction_timeout=None)

Update a custom model validation.

Parameters:

name (Optional[str], optional) – The new name of the custom model validation.
prompt_column_name (Optional[str], optional) – The new name of the prompt column.
target_column_name (Optional[str], optional) – The new name of the target column.
deployment (Optional[Union[Deployment, str]], optional) – The new deployment to validate.
model (Optional[Union[Model, str]], optional) – The new model within the deployment to validate.
prediction_timeout (Optional[int], optional) – The new timeout, in seconds, for the prediction API used in this custom model validation.

Return type:

CustomModelValidation

delete()

Delete the custom model validation.

Return type:: None

class datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation

Validation record checking the ability of the deployment to serve as a custom model LLM.

Variables:

prompt_column_name (str) – The column name the deployed model expect as the input.
target_column_name (str) – The target name that the deployed model will output.
deployment_id (str) – ID of the deployment.
model_id (str) – ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].
validation_status (str) – Can be TESTING, FAILED, or PASSED. Only PASSED is allowed for use.
deployment_access_data (dict, optional) – Data that will be used for accessing deployment prediction server. Only available for deployments that passed validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - first of 2 auth headers for the prediction server. - authorization_header - second of 2 auth headers for the prediction server. - input_type - Either JSON or CSV - the input type that the model expects. - model_type - Target type of the deployed custom model.
tenant_id (str) – Creating user’s tenant ID.
error_message (Optional[str]) – Additional information for errored validation.
deployment_name (Optional[str]) – The name of the deployment that is validated.
user_name (Optional[str]) – The name of the user
use_case_id (Optional[str]) – The ID of the Use Case associated with the validation.
prediction_timeout (int) – The timeout in seconds for the prediction API used in this custom model validation.

class datarobot.models.genai.vector_database.CustomModelVectorDatabaseValidation

Validation record checking the ability of the deployment to serve as a vector database.

Variables:

prompt_column_name (str) – The column name the deployed model expect as the input.
target_column_name (str) – The target name deployed model will output.
deployment_id (str) – ID of the deployment.
model_id (str) – ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].
validation_status (str) – Can be TESTING, FAILED and PASSED. Only PASSED allowed for use.
deployment_access_data (dict, optional) – Data that will be used for accessing deployment prediction server. Only available for deployments that passed validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - first of 2 auth headers for prediction server. - authorization_header - second of 2 auth headers for prediction server. - input_type - Either JSON or CSV - input type model expects. - model_type - Target type of deployed custom model.
tenant_id (str) – Creating user’s tenant ID.
error_message (Optional[str]) – Additional information for errored validation.
deployment_name (Optional[str]) – The name of the deployment that is validated.
user_name (Optional[str]) – The name of the user
use_case_id (Optional[str]) – The ID of the use case associated with the validation.

class datarobot.models.genai.llm_blueprint.LLMBlueprint

Metadata for a DataRobot GenAI LLM blueprint.

Variables:

id (str) – The LLM blueprint ID.
name (str) – The LLM blueprint name.
description (str) – A description of the LLM blueprint.
is_saved (bool) – Whether the LLM blueprint is saved (meaning the settings are locked and blueprint is eligible for use with ComparisonPrompts).
is_starred (bool) – Whether the LLM blueprint is starred.
playground_id (str) – The ID of the playground associated with the LLM blueprint.
llm_id (str or None) – The ID of the LLM type. If not None, this must be one of the IDs returned by LLMDefinition.list for this user.
llm_name (str or None) – The name of the LLM.
llm_settings (dict or None) – The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the external model LLM validation. - external_llm_context_size - The external LLM’s context size, in tokens, for external model LLM blueprints.
creation_date (str) – The date the playground was created.
creation_user_id (str) – The ID of the user creating the playground.
creation_user_name (str) – The name of the user creating the playground.
last_update_date (str) – The date the playground was last updated.
last_update_user_id (str) – The ID of the user who most recently updated the playground.
prompt_type (PromptType) – The prompting strategy for the LLM blueprint. Currently supported options are listed in PromptType.
vector_database_id (str or None) – The ID of the vector database, if any, associated with the LLM blueprint.
vector_database_settings (VectorDatabaseSettings or None) – The settings for the vector database, if any, associated with the LLM blueprint.
vector_database_name (str or None) – The name of the vector database associated with the LLM blueprint, if any.
vector_database_status (str or None) – The status of the vector database, if any, associated with the LLM blueprint.
vector_database_error_message (str or None) – The error message for the vector database, if any, associated with the LLM blueprint.
vector_database_error_resolution (str or None) – The resolution for the vector database error, if any, associated with the LLM blueprint.
custom_model_llm_validation_status (str or None) – The status of the custom model LLM validation if the llm_id is ‘custom-model’.
custom_model_llm_error_message (str or None) – The error message for the custom model LLM, if any.
custom_model_llm_error_resolution (str or None) – The resolution for the custom model LLM error, if any.

classmethod create(playground, name, prompt_type=PromptType.CHAT_HISTORY_AWARE, description='', llm=None, llm_settings=None, vector_database=None, vector_database_settings=None)

Create a new LLM blueprint.

Parameters:

playground (Playground or str) – The playground associated with the created LLM blueprint. Accepts playground or playground ID.
name (str) – The LLM blueprint name.
prompt_type (PromptType, optional) – Prompting type of the LLM blueprint, by default PromptType.CHAT_HISTORY_AWARE.
description (Optional[str]) – An optional description for the LLM blueprint, otherwise null.
llm (LLMDefinition, str, or None, optional) – The LLM to use for the blueprint, either LLMDefinition or LLM ID.
llm_settings (dict or None) – The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.
vector_database (VectorDatabase, str, or None, optional) – The vector database to use with this LLM blueprint, either VectorDatabase or vector database ID.
vector_database_settings (VectorDatabaseSettings or None, optional) – The settings for the vector database, if any.

Returns:

llm_blueprint – The created LLM blueprint.

Return type:

LLMBlueprint

classmethod create_from_llm_blueprint(llm_blueprint, name, description='')

Create a new LLM blueprint from an existing LLM blueprint.

Parameters:

llm_blueprint (LLMBlueprint or str) – The LLM blueprint to use to create the new LLM blueprint. Accepts LLM blueprint or LLM blueprint ID.
name (str) – LLM blueprint name.
description (Optional[str]) – Description of the LLM blueprint, by default “”.

Returns:

llm_blueprint – The created LLM blueprint.

Return type:

LLMBlueprint

classmethod get(llm_blueprint_id)

Retrieve a single LLM blueprint.

Parameters:: llm_blueprint_id (str) – The ID of the LLM blueprint you want to retrieve.
Returns:: llm_blueprint – The requested LLM blueprint.
Return type:: LLMBlueprint

classmethod list(playground=None, llms=None, vector_databases=None, is_saved=None, is_starred=None, sort=None)

Lists all LLM blueprints available to the user. If the playground is specified, then the results are restricted to the LLM blueprints associated with the playground. If the LLMs are specified, then the results are restricted to the LLM blueprints using those LLM types. If vector_databases are specified, then the results are restricted to the LLM blueprints using those vector databases.

Parameters:

playground (Optional[Union[Playground, str]], optional) – The returned LLM blueprints are filtered to those associated with a specific playground if it is specified. Accepts either the entity or the ID.
llms (Optional[list[Union[LLMDefinition, str]]], optional) – The returned LLM blueprints are filtered to those associated with the LLM types specified. Accepts either the entity or the ID.
vector_databases (Optional[list[Union[VectorDatabase, str]]], optional) – The returned LLM blueprints are filtered to those associated with the vector databases specified. Accepts either the entity or the ID.
is_saved (Optional[bool], optional) – The returned LLM blueprints are filtered to those matching is_saved.
is_starred (Optional[bool], optional) – The returned LLM blueprints are filtered to those matching is_starred.
sort (Optional[str]) – Property to sort LLM blueprints by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListLLMBlueprintsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in LLM blueprints being returned in order of creation time descending.

Returns:

playgrounds – A list of playgrounds available to the user.

Return type:

list[Playground]

update(name=None, description=None, llm=None, llm_settings=None, vector_database=None, vector_database_settings=None, is_saved=None, is_starred=None, prompt_type=None, remove_vector_database=False)

Update the LLM blueprint.

Parameters:

name (str or None, optional) – The new name for the LLM blueprint.
description (str or None, optional) – The new description for the LLM blueprint.
llm (Optional[Union[LLMDefinition, str]], optional) – The new LLM type for the LLM blueprint.
llm_settings (Optional[dict], optional) – The new LLM settings for the LLM blueprint. These must match the LLMSettings returned from the LLMDefinition.list method for the LLM type used for this LLM blueprint but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.
vector_database (Optional[Union[VectorDatabase, str]], optional) – The new vector database for the LLM blueprint.
vector_database_settings (Optional[VectorDatabaseSettings], optional) – The new vector database settings for the LLM blueprint.
is_saved (Optional[bool], optional) – The new is_saved attribute for the LLM blueprint (meaning the settings are locked and blueprint is eligible for use with ComparisonPrompts).
is_starred (Optional[bool], optional) – The new setting for whether the LLM blueprint is starred.
prompt_type (PromptType, optional) – The new prompting type of the LLM blueprint.
remove_vector_database (Optional[bool], optional) – Whether to remove the vector database from the LLM blueprint.

Returns:

llm_blueprint – The updated LLM blueprint.

Return type:

LLMBlueprint

delete()

Delete the single LLM blueprint.

Return type:: None

register_custom_model(prompt_column_name=None, target_column_name=None, llm_test_configuration_ids=None)

Create a new CustomModelVersion. This registers a custom model from the LLM blueprint.

Parameters:

prompt_column_name (Optional[str]) – The column name of the prompt text.
target_column_name (Optional[str]) – The column name of the response text.
llm_test_configuration_ids (List[str], optional) – The IDs of the LLM test configurations to execute.

Returns:

custom_model – The registered custom model.

Return type:

CustomModelVersion

class datarobot.models.genai.llm.LLMDefinition

Metadata for a DataRobot GenAI LLM.

Variables:

id (str) – Language model type ID.
name (str) – Language model name.
description (str) – Description of the language model.
vendor (str) – Name of the vendor for this model.
license (str) – License for this model.
supported_languages (str) – Languages supported by this model.
settings (list of LLMSettingDefinition) – Settings for this model
context_size (int) – The context size for this model

classmethod list(use_case=None, as_dict=True)

List all large language models (LLMs) available to the user.

Parameters:: use_case (Optional[UseCase or str], optional) – The returned LLMs, including external LLMs, available for the specified Use Case. Accepts either the entity or the Use CaseID.
Returns:: llms – A list of large language models (LLMs) available to the user.
Return type:: list[LLMDefinition] or list[LLMDefinitionDict]

class datarobot.models.genai.llm.LLMDefinitionDict

class datarobot.models.genai.playground.Playground

Metadata for a DataRobot GenAI playground.

Variables:

id (str) – Playground ID.
name (str) – Playground name.
description (str) – Description of the playground.
use_case_id (str) – Linked use case ID.
creation_date (str) – The date when the playground was created.
creation_user_id (str) – ID of the creating user.
last_update_date (str) – Date when the playground was most recently updated.
last_update_user_id (str) – ID of the user who most recently updated the playground.
saved_llm_blueprints_count (int) – Number of saved LLM blueprints in the playground.
llm_blueprints_count (int) – Number of LLM blueprints in the playground.
user_name (str) – The name of the user who created the playground.

classmethod create(name, description='', use_case=None, copy_insights=None)

Create a new playground.

Parameters:

name (str) – Playground name.
description (Optional[str]) – Description of the playground, by default “”.
use_case (Optional[Union[UseCase, str]], optional) – Use case to link to the created playground.
copy_insights (CopyInsightsRequest, optional) – If present, copies insights from the source playground to the created playground.

Returns:

playground – The created playground.

Return type:

Playground

classmethod get(playground_id)

Retrieve a single playground.

Parameters:: playground_id (str) – The ID of the playground you want to retrieve.
Returns:: playground – The requested playground.
Return type:: Playground

classmethod list(use_case=None, search=None, sort=None)

List all playgrounds available to the user. If the use_case is specified or can be inferred from the Context then the results are restricted to the playgrounds associated with the UseCase.

Parameters:

use_case (Optional[UseCaseLike], optional) – The returned playgrounds are filtered to those associated with a specific Use Case or Cases if specified or can be inferred from the Context. Accepts either the entity or the ID.
search (Optional[str]) – String for filtering playgrounds. Playgrounds that contain the string in name will be returned. If not specified, all playgrounds will be returned.
sort (Optional[str]) – Property to sort playgrounds by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListPlaygroundsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in playgrounds being returned in order of creation time descending.

Returns:

playgrounds – A list of playgrounds available to the user.

Return type:

list[Playground]

update(name=None, description=None)

Update the playground.

Parameters:

name (str) – The new name for the playground.
description (str) – The new description for the playground.

Returns:

playground – The updated playground.

Return type:

Playground

delete()

Delete the playground.

Return type:: None

class datarobot.enums.PromptType: Supported LLM blueprint prompting types.

class datarobot.models.genai.vector_database.SupportedEmbeddings

All supported embedding models including the recommended default model.

Variables:

embedding_models (list[EmbeddingModel]) – All supported embedding models.
default_embedding_model (str) – Name of the default recommended text embedding model. Currently supported options are listed in VectorDatabaseEmbeddingModel but the values can differ with different platform versions.
custom_model_embedding_validations (List[str]) – External embedding models that have been validated

class datarobot.models.genai.vector_database.SupportedTextChunkings

Supported text chunking configurations which includes a set of recommended chunking parameters for each supported embedding model.

Variables:: text_chunking_configs – All supported text chunking configurations.

class datarobot.models.genai.user_limits.UserLimits

Counts for user limits for LLM APIs and vector databases.

classmethod get_vector_database_count()

Get the count of vector databases for the user.

Return type:: APIObject

classmethod get_llm_requests_count()

Get the count of LLMs requests made by the user.

Return type:: APIObject

class datarobot.models.genai.vector_database.VectorDatabase

Metadata for a DataRobot vector database accessible to the user.

Variables:

id (str) – Vector database ID.
name (str) – Vector database name.
size (int) – Size of the vector database assets in bytes.
use_case_id (str) – Linked use case ID.
dataset_id (str) – ID of the dataset used for creation.
embedding_model (str) – Name of the text embedding model. Currently supported options are listed in VectorDatabaseEmbeddingModel but the values can differ with different platform versions.
chunking_method (str or None) – Name of the method to split dataset documents. Currently supported options are listed in VectorDatabaseChunkingMethod but the values can differ with different platform versions.
chunk_size (int or None) – Size of each text chunk in number of tokens.
chunk_overlap_percentage (int or None) – Overlap percentage between chunks.
chunks_count (int) – Total number of text chunks.
custom_chunking (bool) – Determines if the chunking is custom. With custom chunking, dataset rows are not split into chunks automatically; instead, the user provides the chunks.
separators (list[string] or None) – Separators for document splitting.
creation_date (str) – Date when the database was created.
creation_user_id (str) – ID of the creating user.
organization_id (str) – Creating user’s organization ID.
tenant_id (str) – Creating user’s tenant ID.
last_update_date (str) – Last update date for the database.
execution_status (str) – Database execution status. Currently supported options are listed in VectorDatabaseExecutionStatus but the values can differ with different platform versions.
playgrounds_count (int) – Number of using playgrounds.
dataset_name (str) – Name of the used dataset.
user_name (str) – Name of the creating user.
source (str) – Source of the vector database. Currently supported options are listed in VectorDatabaseSource but the values can differ with different platform versions.
validation_id (Optional[str]) – ID of custom model vector database validation. Only filled for external vector databases.
error_message (Optional[str]) – Additional information for errored vector database.
embedding_validation_id (Optional[str]) – ID of the custom embedding validation, if any.
is_separator_regex (bool) – Whether the separators should be treated as regular expressions.

classmethod get_supported_embeddings(dataset_id=None, use_case=None)

Get all supported and the recommended embedding models.

Parameters:

dataset_id (Optional[str]) – ID of a dataset for which the recommended model is returned based on the detected language of that dataset.
use_case (Optional[UseCase, str]) – May be Use Case ID or the Use Case entity.

Returns:

supported_embeddings – The supported embedding models.

Return type:

SupportedEmbeddings

submit_export_dataset_job()

Submit the vector database dataset export job.

Returns:: result – The result of the vector database dataset export job containing the exported dataset id.
Return type:: VectorDatabaseDatasetExportJob

classmethod get_supported_retrieval_settings()

Get supported retrieval settings.

Returns:: supported_retrieval_settings – The supported retriever settings.
Return type:: SupportedRetrievalSettings

classmethod create(dataset_id, chunking_parameters=None, use_case=None, name=None, parent_vector_database_id=None, update_llm_blueprints=None, update_deployments=None)

Create a new vector database.

Parameters:

dataset_id (str) – ID of the dataset used for creation.
chunking_parameters (ChunkingParameters) – Parameters defining how documents are split and embedded.
use_case (Optional[Union[UseCase, str]], optional) – Use case to link to the created vector database.
name (Optional[str]) – Vector database name, by default None which leads to the default name ‘Vector Database for <dataset name>’.
parent_vector_database_id (Optional[str]) – ID of the parent vector database to base the update on.
update_llm_blueprints (Optional[bool]) – Whether to update LLM blueprints related to the parent vector database.
update_deployments (Optional[bool]) – Whether to update deployments related to the parent vector database.

Returns:

vector database – The created vector database with execution status ‘new’.

Return type:

VectorDatabase

classmethod create_from_custom_model(name, use_case=None, validation_id=None, prompt_column_name=None, target_column_name=None, deployment_id=None, model_id=None)

Create a new vector database from validated custom model deployment.

Parameters:

name (str) – Vector database name.
use_case (Optional[Union[UseCase, str]], optional) – Use case to link to the created vector database.
validation_id (Optional[str]) – ID of CustomModelVectorDatabaseValidation for the deployment. Alternatively, you can specify ALL the following fields.
prompt_column_name (Optional[str]) – The column name the deployed model expect as the input.
target_column_name (Optional[str]) – The target name deployed model will output.
deployment_id (Optional[str]) – ID of the deployment.
model_id (Optional[str]) – ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].

Returns:

vector database – The created vector database.

Return type:

VectorDatabase

classmethod get(vector_database_id)

Retrieve a single vector database.

Parameters:: vector_database_id (str) – The ID of the vector database you want to retrieve.
Returns:: vector database – The requested vector database.
Return type:: VectorDatabase

classmethod list(use_case=None, playground=None, search=None, sort=None, completed_only=None)

List all vector databases associated with a specific use case available to the user.

Parameters:

use_case (Optional[UseCaseLike], optional) – The returned vector databases are filtered to those associated with a specific Use Case or Cases if specified or can be inferred from the Context. Accepts either the entity or the ID.
playground (Optional[Union[Playground, str]], optional) – The returned vector databases are filtered to those associated with a specific playground if it is specified. Accepts either the entity or the ID.
search (Optional[str]) – String for filtering vector databases. Vector databases that contain the string in name will be returned. If not specified, all vector databases will be returned.
sort (Optional[str]) – Property to sort vector databases by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListVectorDatabasesSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in vector databases being returned in order of creation time descending.
completed_only (Optional[bool]) – A filter to retrieve only vector databases that have been successfully created. By default, all vector databases regardless of execution status are retrieved.

Returns:

vectorbases – A list of vector databases available to the user.

Return type:

list[VectorDatabase]

update(name)

Update the vector database.

Parameters:: name (str) – The new name for the vector database.
Returns:: vector database – The updated vector database.
Return type:: VectorDatabase

delete()

Delete the vector database.

Return type:: None

classmethod get_supported_text_chunkings()

Get all supported text chunking configurations which includes a set of recommended chunking parameters for each supported embedding model.

Returns:: supported_text_chunkings – The supported text chunking configurations.
Return type:: SupportedTextChunkings

download_text_and_embeddings_asset(file_path=None)

Download a parquet file with text chunks and corresponding embeddings created by a vector database.

Parameters:: file_path (Optional[str]) – File path to save the asset. By default, it saves in the current directory autogenerated by server name.
Return type:: None

class datarobot.models.genai.vector_database.SupportedRetrievalSetting

A single supported retrieval setting.

Variables:

name (str) – The name of the setting.
type (str or list[str]) – The type of the setting.
description (str) – The description of the setting.
title (str) – The title of the setting.
default (str, int, bool, or None) – The default value of the setting.
minimum (int or None) – The minimum value of the setting.
maximum (int or None) – The maximum value of the setting.
enum (list[str] or None) – The enum values of the setting.
settings (list[SupportedRetrievalSetting] or None) – The supported retriever settings.
group_id (str or None) – The group ID of the setting.

class datarobot.models.genai.vector_database.SupportedRetrievalSettings

Supported retrieval settings.

Variables:: settings (list[dict]) – The supported retriever settings.

class datarobot.models.genai.vector_database.VectorDatabaseDatasetExportJob

Response for the vector database dataset export job.

Variables:

job_id (str) – ID of the export job.
vector_database_id (str) – ID of the vector database.
export_dataset_id (str) – ID of the exported dataset.

class datarobot.models.genai.chat_prompt.Citation

Citation for documents retrieved from a vector database.

Variables:

text (str) – The text retrieved from a vector database.
source (str or None, optional) – The source of the retrieved text.
similarity_score – The similarity score between the citation and the user prompt.
metadata (dict or None, optional) – Additional metadata for the citation.

class datarobot.models.genai.chat_prompt.ResultMetadata

Metadata for the result of a chat prompt submission.

Variables:

output_token_count (int) – The number of tokens in the output.
input_token_count (int) – The number of tokens in the input. This includes the chat history and documents retrieved from a vector database, if any.
total_token_count (int) – The total number of tokens processed.
estimated_docs_token_count (int) – The estimated number of tokens from the documents retrieved from a vector database, if any.
latency_milliseconds (int) – The latency of the chat prompt submission in milliseconds.
feedback_result (FeedbackResult) – The lists of user_ids providing positive and negative feedback.
metrics (MetricMetadata) – The evaluation metrics for this prompt.
final_prompt (Optional[Union[str, dict]], optional) – Representation of the final prompt sent to the LLM.
error_message (str or None, optional) – The error message from the LLM response.
cost (float or None, optional) – The cost of the chat prompt submission.

class datarobot.models.genai.llm_blueprint.VectorDatabaseSettings

Settings for a DataRobot GenAI vector database associated with an LLM blueprint.

Variables:

max_documents_retrieved_per_prompt (int or None, optional) – The maximum number of documents to retrieve for each prompt.
max_tokens (int or None, optional) – The maximum number of tokens to retrieve for each document.
retriever (VectorDatabaseRetrievers) – The vector database retriever name.
add_neighbor_chunks – Whether to add neighboring documents to the retrieved documents.

class datarobot.models.genai.vector_database.ChunkingParameters

Parameters defining how documents are split and embedded.

Variables:

embedding_model (Optional[str]) – Name of the text embedding model. Currently supported options are listed in VectorDatabaseEmbeddingModel but the values can differ with different platform versions.
chunking_method (str) – Name of the method to split dataset documents. Currently supported options are listed in VectorDatabaseChunkingMethod but the values can differ with different platform versions.
chunk_size (int) – Size of each text chunk in number of tokens.
chunk_overlap_percentage (int) – Overlap percentage between chunks.
separators (list[str]) – Strings used to split documents into text chunks.
embedding_validation (Optional[CustomModelEmbeddingValidation, SupportedCustomModelEmbedding, str]) – ID or object for custom embedding validation.
custom_chunking (bool) – Determines if the chunking is custom. With custom chunking, dataset rows are not split into chunks automatically; instead, the user provides the chunks.

class datarobot.models.genai.prompt_trace.PromptTrace

Prompt trace contains aggregated information about a prompt execution.

Variables:

timestamp (str) – The timestamp of the trace (ISO 8601 formatted).
user (dict) – The user who submitted the prompt.
chat_prompt_id (str) – The ID of the chat prompt associated with the trace.
use_case_id (str) – The ID of the Use Case the playground is in.
comparison_prompt_id (str) – The ID of the comparison prompts associated with the trace.
llm_blueprint_id (str) – The ID of the LLM blueprint that the prompt was submitted to.
llm_blueprint_name (str) – The name of the LLM blueprint.
llm_name (str) – The name of the LLM in the LLM blueprint.
llm_vendor (str) – The vendor name of the LLM.
llm_license (str) – What type of license the LLM has.
llm_settings (dict or None) – The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list, but this typically has dict fields. Either: - system_prompt - The system prompt that influences the LLM responses. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Sets whether the model considers next tokens with top_p probability mass. or - system_prompt - The system prompt that influences the LLM responses. - validation_id - The ID of the external model LLM validation. - external_llm_context_size - The external LLM’s context size, in tokens, for external model-based LLM blueprints.
chat_name (str or None) – The name of the chat associated with the Trace.
chat_id (str or None) – The ID of the chat associated with the Trace.
vector_database_id (str or None) – ID of the vector database associated with the LLM blueprint, if any.
vector_database_settings (VectorDatabaseSettings or None) – The settings for the vector database associated with the LLM blueprint, if any.
result_metadata (ResultMetadata or None) – Metadata for the result of the prompt submission.
result_text (str or None) – The result text from the prompt submission.
confidence_scores (ConfidenceScores or None) – The confidence scores if there is a vector database associated with the prompt.
text (str) – The prompt text submitted to the LLM.
execution_status (str) – The execution status of the chat prompt.
prompt_type (str or None) – The type of prompting strategy, for example history aware.
evaluation_dataset_configuration_id (str or None) – The ID of the evaluation dataset configuration associated with the trace.
warning (str or None) – Any warnings associated with the trace.

classmethod list(playground)

List all prompt traces for a playground.

Parameters:: playground (str) – The ID of the playground to list prompt traces for.
Returns:: prompt_traces – List of prompt traces for the playground.
Return type:: list[PromptTrace]

classmethod export_to_ai_catalog(playground)

Export prompt traces to AI Catalog as a CSV.

Parameters:: playground (str) – The ID of the playground to export prompt traces for.
Returns:: status_url – The URL where the status of the job can be monitored
Return type:: str

class datarobot.models.genai.insights_configuration.InsightsConfiguration

Configuration information for a specific insight.

Variables:

insight_name (str) – The name of the insight.
insight_type (InsightTypes, optional) – The type of the insight.
deployment_id (Optional[str]) – The deployment ID the insight is applied to.
model_id (Optional[str]) – The model ID for the insight.
sidecar_model_metric_validation_id (Optional[str]) – Validation ID for the sidecar model metric.
custom_metric_id (Optional[str]) – The ID for a custom model metric.
evaluation_dataset_configuration_id (Optional[str]) – The ID for the evaluation dataset configuration.
cost_configuration_id (Optional[str]) – The ID for the cost configuration information.
result_unit (Optional[str]) – The unit of the result, for example “USD”.
ootb_metric_id (Optional[str]) – The ID of the Datarobot-provided metric that does not require additional configuration.
ootb_metric_name (Optional[str]) – The name of the Datarobot-provided metric that does not require additional configuration.
guard_conditions (list[dict], optional) – The guard conditions to be used with the insight.
moderation_configuration (dict, optional) – The moderation configuration for the insight.
execution_status (Optional[str]) – The execution status of the insight.
error_message (Optional[str]) – The error message for the insight, for example if it is missing specific configuration for deployed models.
error_resolution (Optional[str]) – An indicator of which field must be edited to resolve an error state.
nemo_metric_id (Optional[str]) – The ID for the NEMO metric.
llm_id (Optional[str]) – The LLM ID for OOTB metrics that use LLMs.
custom_model_llm_validation_id (Optional[str]) – The ID for the custom model LLM validation if using a custom model LLM for OOTB metrics.
aggregation_types (list[str], optional) – The aggregation types to be used for the insight.
stage (Optional[str]) – The stage (prompt or response) when the metric is calculated.
sidecar_model_metric_metadata (dict, optional) – Metadata specific to sidecar model metrics.
guard_template_id (Optional[str]) – The ID for the guard template that applies to the insight.
guard_configuration_id (Optional[str]) – The ID for the guard configuration that applies to the insight.

class datarobot.models.genai.insights_configuration.SupportedInsights

Supported insights configurations for a given use case.

classmethod list(use_case_id)

Get a list of all supported insights that can be used within a given Use Case.

Parameters:: use_case_id (str) – The ID of the Use Case to list supported insights for.
Returns:: insights – A list of supported insights.
Return type:: list[InsightsConfiguration]

class datarobot.models.genai.insights_configuration.Insights

The insights configured for a playground.

Variables:

playground_id (str) – The ID of the playground the insights are configured for.
insights_configuration (list[InsightsConfiguration]) – The insights configuration for the playground.
creation_date (str) – The date the insights were configured.
creation_user_id (str) – The ID of the user who created the insights.
last_update_date (str) – The date the insights were last updated.
last_update_user_id (str) – The ID of the user who last updated the insights.
tenant_id (str) – The tenant ID that applies to the record.

classmethod get(playground, with_aggregation_types_only=False)

Get the insights configuration for a given playground.

Parameters:

playground (str|Playground) – The ID of the playground to get insights for.
with_aggregation_types_only (Optional[bool]) – If True, only return the aggregation types for the insights.

Returns:

insights – The insights configuration for the playground.

Return type:

Insights

classmethod create(playground, insights_configuration, use_case)

Create a new insights configuration for a given playground.

Parameters:

playground (str) – The ID of the playground to create insights for.
insights_configuration (list[InsightsConfiguration]) – The insights configuration for the playground.
use_case_id (str) – The Use Case ID to the playground is a part of.

Returns:

insights – The created insights configuration.

Return type:

Insights

class datarobot.models.genai.cost_metric_configurations.LLMCostConfiguration

Cost configuration for a specific LLM model; used for cost metric calculation. Price-per-token is price/reference token count.

Variables:

(float) (output_token_price)
(int) (reference_output_token_count)
(float)
(int)
(str) (llm_id)
(str)
(Optional[str]) (custom_model_llm_validation_id)

class datarobot.models.genai.cost_metric_configurations.CostMetricConfiguration

Cost metric configuration for a use case.

Variables:

(str) (use_case_id)
(str)
(List[LLMCostConfiguration]) (cost_metric_configurations)

classmethod get(cost_metric_configuration_id)

Get cost metric configuration by ID.

Return type:: CostMetricConfiguration

update(cost_metric_configurations, name=None)

Update the cost configurations.

Return type:: CostMetricConfiguration

classmethod create(use_case_id, playground_id, name, cost_metric_configurations)

Create a new cost metric configuration.

Return type:: CostMetricConfiguration

delete()

Delete the cost metric configuration.

Return type:: None

class datarobot.models.genai.evaluation_dataset_configuration.EvaluationDatasetConfiguration

An evaluation dataset configuration used to evaluate the performance of LLMs.

Variables:

id (str) – The evaluation dataset configuration ID.
name (str) – The name of the evaluation dataset configuration.
size (int) – The size of the evaluation dataset (in bytes).
rows_count (int) – The row count of the evaluation dataset.
use_case_id (str) – The ID of the Use Case associated with the evaluation dataset configuration.
playground_id (Optional[str]) – The ID of the playground associated with the evaluation dataset configuration.
dataset_id (str) – The ID of the evaluation dataset.
dataset_name (str) – The name of the evaluation dataset.
prompt_column_name (str) – The name of the dataset column containing the prompt text.
response_column_name (Optional[str]) – The name of the dataset column containing the response text.
user_name (str) – The name of the user who created the evaluation dataset configuration.
correctness_enabled (Optional[bool]) – Whether correctness is enabled for the evaluation dataset configuration.
creation_user_id (str) – The ID of the user who created the evaluation dataset configuration.
creation_date (str) – The creation date of the evaluation dataset configuration (ISO-8601 formatted).
tenant_id (str) – The ID of the DataRobot tenant this evaluation dataset configuration belongs to.
execution_status (str) – The execution status of the evaluation dataset configuration.
error_message (Optional[str]) – The error message associated with the evaluation dataset configuration.

classmethod get(id)

Get an evaluation dataset configuration by ID.

Parameters:: id (str) – The evaluation dataset configuration ID to fetch.
Returns:: evaluation_dataset_configuration – The evaluation dataset configuration.
Return type:: EvaluationDatasetConfiguration

classmethod list(use_case_id, playground_id, evaluation_dataset_configuration_id=None, offset=0, limit=100, sort=None, search=None, correctness_only=False, completed_only=False)

List all evaluation dataset configurations for a Use Case.

Parameters:

use_case_id (str) – The ID of the Use Case that evaluation datasets are returned for.
playground_id (str) – The ID of the playground that evaluation datasets are returned for. Default is None.
evaluation_dataset_configuration_id (Optional[str]) – The ID of the evaluation dataset configuration to fetch. Default is None.
offset (Optional[int]) – The offset to start fetching evaluation datasets from. Default is 0.
limit (Optional[int]) – The maximum number of evaluation datasets to return. Default is 100.
sort (Optional[str]) – The order of return for evaluation datasets. Default is None, which returns sorting by creation time.
search (Optional[str]) – A search term that filters results so that only evaluation datasets with names matching the string are returned. Default is None.
correctness_only (Optional[bool]) – Whether to return only completed datasets (particularly applicable to completion of generated synthetic datasets). Default is False.
completed_only (Optional[bool]) – Whether to return only completed datasets. Default is False.

Returns:

evaluation_dataset_configurations – A list of evaluation dataset configurations.

Return type:

List[EvaluationDatasetConfiguration]

classmethod create(name, use_case_id, dataset_id, prompt_column_name, playground_id, is_synthetic_dataset=False, response_column_name=None)

Create an evaluation dataset configuration for an existing dataset.

Parameters:

name (str) – The name of the evaluation dataset configuration.
use_case_id (str) – The Use Case ID that the evaluation dataset configuration will be added to.
dataset_id (str) – An ID, to add to the configuration, that identifies the evaluation dataset.
playground_id (str) – The ID of the playground that the evaluation dataset configuration will be added to. Default is None.
prompt_column_name (str) – The name of the prompt column in the dataset.
response_column_name (str) – The name of the response column in the dataset.
is_synthetic_dataset (bool) – Whether the evaluation dataset is synthetic.

Returns:

evaluation_dataset_configuration – The created evaluation dataset configuration.

Return type:

EvaluationDatasetConfiguration

update(name=None, dataset_id=None, prompt_column_name=None, response_column_name=None)

Update the evaluation dataset configuration.

Parameters:

name (Optional[str]) – The name of the evaluation dataset configuration.
dataset_id (Optional[str]) – The ID of the dataset used in this configuration.
prompt_column_name (Optional[str]) – The name of the prompt column in the dataset.
response_column_name (Optional[str]) – The name of the response column in the dataset.

Returns:

evaluation_dataset_configuration – The updated evaluation dataset configuration.

Return type:

EvaluationDatasetConfiguration

delete()

Delete the evaluation dataset configuration.

Return type:: None

class datarobot.models.genai.evaluation_dataset_metric_aggregation.EvaluationDatasetMetricAggregation

Information about an evaluation dataset metric aggregation job.: This job runs a metric against LLMs using an evaluation dataset and aggregates the results.

Variables:

llm_blueprint_id (str) – The LLM blueprint ID.
evaluation_dataset_configuration_id (str) – The evaluation dataset configuration ID.
ootb_dataset_name (str | None) – The name of the Datarobot-provided dataset that does not require additional configuration..
metric_name (str) – The name of the metric.
deployment_id (str | None) – A deployment ID if the evaluation was run against a deployment.
dataset_id (str | None) – The ID of the dataset used in the evaluation.
dataset_name (str | None) – The name of the dataset used in the evaluation.
chat_id (str) – The ID of the chat created to run the evaluation.
chat_name (str) – The name of the chat that was created to run the evaluation.
aggregation_value (float | List[Dict[str, float]]) – The aggregated metric result.
aggregation_type (AggregationType) – The type of aggregation used for the metric results.
creation_date (str) – The date the evaluation job was created.
creation_user_id (str) – The ID of the user who created the evaluation job.
tenant_id (str) – The ID of the tenant that owns the evaluation job.

classmethod create(chat_name, llm_blueprint_ids, evaluation_dataset_configuration_id, insights_configuration)

Create a new evaluation dataset metric aggregation job. The job will run the specified metric for the specified LLM blueprint IDs using the prompt-response pairs in the evaluation dataset.

Parameters:

chat_name (str) – The name of the chat that will be created to run the evaluation in.
llm_blueprint_ids (List[str]) – The LLM blueprint IDs to evaluate.
evaluation_dataset_configuration_id (str) – The ID evaluation dataset configuration to use during the evaluation.
insights_configuration (List[InsightsConfiguration]) – The insights configurations to use during the evaluation.

Returns:

The ID of the evaluation dataset metric aggregation job.

Return type:

str

classmethod list(llm_blueprint_ids=None, chat_ids=None, evaluation_dataset_configuration_ids=None, metric_names=None, aggregation_types=None, current_configuration_only=False, sort=None, offset=0, limit=100, non_errored_only=True)

List evaluation dataset metric aggregations. The results will be filtered by the provided LLM blueprint IDs and chat IDs.

Parameters:

llm_blueprint_ids (List[str]) – The LLM blueprint IDs to filter on.
chat_ids (List[str]) – The chat IDs to filter on.
evaluation_dataset_configuration_ids (List[str]) – The evaluation dataset configuration IDs to filter on.
metric_names (List[str]) – The metric names to filter on.
aggregation_types (List[str]) – The aggregation types to filter on.
current_configuration_only (Optional[bool]) – If True, only results that are associated with the current configuration of the LLM blueprint will be returned. Defaults to False.
sort (Optional[str]) – The field to sort on. Defaults to None.
offset (Optional[int]) – The offset to start at. Defaults to 0.
limit (Optional[int]) – The maximum number of results to return. Defaults to 100.
non_errored_only (Optional[bool]) – If True, only results that did not encounter an error will be returned. Defaults to True.

Returns:

A list of evaluation dataset metric aggregations.

Return type:

List[EvaluationDatasetMetricAggregation]

classmethod delete(llm_blueprint_ids, chat_ids)

Delete the associated evaluation dataset metric aggregations. Either llm_blueprint_ids or chat_ids must be provided. If both are provided, only results matching both will be removed.

Parameters:

llm_blueprint_ids (List[str]) – The LLM blueprint IDs to filter on.
chat_ids (List[str]) – The chat IDs to filter on.

Return type:

None

class datarobot.models.genai.synthetic_evaluation_dataset_generation.SyntheticEvaluationDataset

A synthetically generated evaluation dataset for LLMs.

Variables:

(str) (response_column_name)
(str)
(str)

classmethod create(llm_id, vector_database_id, llm_settings=None, dataset_name=None, language=None)

Create a synthetic evaluation dataset generation job. This will create a synthetic dataset to be used for evaluation of a language model.

Parameters:

(str) (language)
(Dict[Optional[str][Union[bool (llm_settings) – model used for dataset generation.
int (The settings to use for the language) – model used for dataset generation.
float (The settings to use for the language) – model used for dataset generation.
str]]]) (The settings to use for the language) – model used for dataset generation.
(str)
(str)
(str)

Returns:

SyntheticEvaluationDataset

Return type:

Reference to the synthetic evaluation dataset that was created.

class datarobot.models.genai.sidecar_model_metric.SidecarModelMetricValidation

A sidecar model metric validation for LLMs.

Variables:

id (str) – The ID of the sidecar model metric validation.
prompt_column_name (str) – The name of the prompt column for the sidecar model.
deployment_id (str) – The ID of the deployment associated with the sidecar model.
model_id (str) – The ID of the sidecar model.
validation_status (str) – The status of the validation job.
deployment_access_data (dict) – Data that will be used for accessing deployment prediction server. Only available for deployments that passed validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - first of 2 auth headers for the prediction server. - authorization_header - second of 2 auth headers for the prediction server. - input_type - Either JSON or CSV - the input type that the model expects. - model_type - Target type of the deployed custom model.
tenant_id (str) – The ID of the tenant that created the sidecar model metric validation.
name (str) – The name of the sidecar model metric.
creation_date (str) – The date the sidecar model metric validation was created.
user_id (str) – The ID of the user that created the sidecar model metric validation.
deployment_name (str) – The name of the deployment associated with the sidecar model.
user_name (str) – The name of the user that created the sidecar model metric validation.
use_case_id (str) – The ID of the use case associated with the sidecar model metric validation.
prediction_timeout (int) – The timeout in seconds for the prediction API used in this sidecar model metric validation.
error_message (str) – Additional information for errored validation.
citations_prefix_column_name (str) – The name of the prefix in the citations column for the sidecar model.
response_column_name (str) – The name of the response column for the sidecar model.
expected_response_column_name (str) – The name of the expected response column for the sidecar model.
target_column_name (str) – The name of the target column for the sidecar model.

classmethod create(deployment_id, name, prediction_timeout, model_id=None, use_case_id=None, playground_id=None, prompt_column_name=None, target_column_name=None, response_column_name=None, citation_prefix_column_name=None, expected_response_column_name=None)

Create a sidecar model metric validation.

Parameters:

deployment_id (str) – The ID of the deployment to validate.
name (str) – The name of the validation.
prediction_timeout (int) – The timeout in seconds for the prediction API used in this validation.
model_id (Optional[str]) – The ID of the model to validate.
use_case_id (Optional[str]) – The ID of the use case associated with the validation.
playground_id (Optional[str]) – The ID of the playground associated with the validation.
prompt_column_name (Optional[str]) – The name of the prompt column for the sidecar model.
target_column_name (Optional[str]) – The name of the target column for the sidecar model.
response_column_name (Optional[str]) – The name of the response column for the sidecar model.
citation_prefix_column_name (Optional[str]) – The name of the prefix for citations column for the sidecar model.
expected_response_column_name (Optional[str]) – The name of the expected response column for the sidecar model.

Returns:

The created sidecar model metric validation.

Return type:

SidecarModelMetricValidation

classmethod list(use_case_ids=None, offset=None, limit=None, search=None, sort=None, completed_only=True, deployment_id=None, model_id=None, prompt_column_name=None, target_column_name=None, citation_prefix_column_name=None)

List sidecar model metric validations.

Parameters:

use_case_ids (List[str], optional) – The IDs of the use cases to filter by.
offset (Optional[int]) – The number of records to skip.
limit (Optional[int]) – The maximum number of records to return.
search (Optional[str]) – The search string.
sort (Optional[str]) – The sort order.
completed_only (Optional[bool]) – Whether to return only completed validations.
deployment_id (Optional[str]) – The ID of the deployment to filter by.
model_id (Optional[str]) – The ID of the model to filter by.
prompt_column_name (Optional[str]) – The name of the prompt column to filter by.
target_column_name (Optional[str]) – The name of the target column to filter by.
citation_prefix_column_name (Optional[str]) – The name of the prefix for citations column to filter by.

Returns:

The list of sidecar model metric validations.

Return type:

List[SidecarModelMetricValidation]

classmethod get(validation_id)

Get a sidecar model metric validation by ID.

Parameters:: validation_id (str) – The ID of the validation to get.
Returns:: The sidecar model metric validation.
Return type:: SidecarModelMetricValidation

revalidate()

Revalidate the sidecar model metric validation.

Returns:: The sidecar model metric validation.
Return type:: SidecarModelMetricValidation

update(name=None, prompt_column_name=None, target_column_name=None, response_column_name=None, expected_response_column_name=None, citation_prefix_column_name=None, deployment_id=None, model_id=None, prediction_timeout=None)

Update the sidecar model metric validation.

Parameters:

name (Optional[str]) – The name of the validation.
prompt_column_name (Optional[str]) – The name of the prompt column for the sidecar model.
target_column_name (Optional[str]) – The name of the target column for the sidecar model.
response_column_name (Optional[str]) – The name of the response column for the sidecar model.
expected_response_column_name (Optional[str]) – The name of the expected response column for the sidecar model.
citation_prefix_column_name (Optional[str]) – The name of the prefix for citations column for the sidecar model.
deployment_id (Optional[str]) – The ID of the deployment to validate.
model_id (Optional[str]) – The ID of the model to validate.
prediction_timeout (Optional[int]) – The timeout in seconds for the prediction API used in this validation.

Returns:

The updated sidecar model metric validation.

Return type:

SidecarModelMetricValidation

delete()

Delete the sidecar model metric validation.

Return type:: None

class datarobot.models.genai.llm_test_configuration.LLMTestConfiguration

Metadata for a DataRobot GenAI LLM test configuration.

Variables:

id (str) – The LLM test configuration ID.
name (str) – The LLM test configuration name.
description (str) – The LLM test configuration description.
dataset_evaluations (list[DatasetEvaluation]) – The dataset/insight combinations that make up the LLM test configuration.
llm_test_grading_criteria (LLMTestGradingCriteria) – The criteria used to grade the result of the LLM test configuration.
is_out_of_the_box_test_configuration (bool) – Whether this is an out-of-the-box configuration.
use_case_id (Optional[str]) – The ID of the linked Use Case, if any.
creation_date (Optional[str]) – The date the LLM test configuration was created, if any.
creation_user_id (Optional[str]) – The ID of the creating user, if any.
warnings (Optional[list[Dict[str, str]]]) – The warnings for the LLM test configuration, if any.

classmethod create(name, dataset_evaluations, llm_test_grading_criteria, use_case=None, description=None)

Creates a new LLM test configuration.

Parameters:

name (str) – The LLM test configuration name.
dataset_evaluations (list[DatasetEvaluationRequestDict]) – The LLM test dataset evaluation requests.
llm_test_grading_criteria (LLMTestGradingCriteria) – The LLM test grading criteria.
use_case (Optional[Union[UseCase, str]], optional) – Use case to link to the created llm test configuration.
description (Optional[str]) – The LLM test configuration description. If None, the default, description returns an empty string.

Returns:

llm_test_configuration – The created LLM test configuration.

Return type:

LLMTestConfiguration

classmethod get(llm_test_configuration)

Retrieve a single LLM Test configuration.

Parameters:: llm_test_configuration (LLMTestConfiguration or str) – The LLM test configuration to retrieve, either LLMTestConfiguration or LLMTestConfiguration ID.
Returns:: llm_test_configuration – The requested LLM Test configuration.
Return type:: LLMTestConfiguration

classmethod list(use_case=None, test_config_type=None)

List all LLM test configurations available to the user. If a Use Case is specified, results are restricted to only those configurations associated with that Use Case.

Parameters:

use_case (Optional[UseCaseLike], optional) – Returns only those configurations associated with a particular Use Case, specified by either the Use Case name or ID.
test_config_type (Optional[LLMTestConfigurationType], optional) – Returns only configurations of the specified type. If not specified, the custom test configurations are returned.

Returns:

llm_test_configurations – Returns a list of LLM test configurations.

Return type:

list[LLMTestConfiguration]

update(name=None, description=None, dataset_evaluations=None, llm_test_grading_criteria=None)

Update the LLM test configuration.

Parameters:

name (Optional[str]) – The new LLM test configuration name.
description (Optional[str]) – The new LLM test configuration description.
dataset_evaluations (list[DatasetEvaluationRequestDict], optional) – The new dataset evaluation requests.
llm_test_grading_criteria (LLMTestGradingCriteria, optional) – The new grading criteria.

Returns:

llm_test_configuration – The updated LLM test configuration.

Return type:

LLMTestConfiguration

delete()

Delete a single LLM test configuration.

Return type:: None

class datarobot.models.genai.llm_test_configuration.LLMTestConfigurationSupportedInsights

Metadata for a DataRobot GenAI LLM test configuration supported insights.

Variables:: supported_insight_configurations (list[InsightsConfiguration]) – The supported insights for LLM test configurations.

classmethod list(use_case=None, playground=None)

List all supported insights for a LLM test configuration.

Parameters:

use_case (Optional[Union[UseCase, str]], optional) – Returns only those supported insight configurations associated with a particular Use Case, specified by either the Use Case name or ID.
playground (Optional[Union[Playground, str]], optional) – Returns only those supported insight configurations associated with a particular playground, specified by either the Playground or ID.

Returns:

llm_test_configuration_supported_insights – Returns the supported insight configurations for the LLM test configuration.

Return type:

LLMTestConfigurationSupportedInsights

class datarobot.models.genai.llm_test_result.LLMTestResult

Metadata for a DataRobot GenAI LLM test result.

Variables:

id (str) – The LLM test result ID.
llm_test_configuration_id (str) – The LLM test configuration ID associated with this LLM test result.
llm_test_configuration_name (str) – The LLM test configuration name associated with this LLM test result.
use_case_id (str) – The ID of the Use Case associated with this LLM test result.
llm_blueprint_id (str) – The ID of the LLM blueprint for this LLM test result.
llm_test_grading_criteria (LLMTestGradingCriteria) – The criteria used to grade the result of the LLM test configuration.
grading_result (GradingResult) – The overall grading result for the LLM test.
pass_percentage (float) – The percentage of insight evaluation results that passed the grading criteria.
execution_status (str) – The execution status of the job that evaluated the LLM test result.
insight_evaluation_result (list[InsightEvaluationResult]) – The results for the individual insights that make up the LLM test result.
creation_date (str) – The date of the LLM test result.
creation_user_id (str) – The ID of the user who executed the LLM test.
creation_user_name (str) – The name of the user who executed the LLM test.

classmethod create(llm_test_configuration, llm_blueprint)

Create a new LLMTestResult. This executes the LLM test configuration using the specified LLM blueprint. To check the status of the LLM test, use the LLMTestResult.get method with the returned ID.

Parameters:

llm_test_configuration (LLMTestConfiguration or str) – The LLM test configuration to execute, either LLMTestConfiguration or the LLM test configuration ID.
llm_blueprint (LLMBlueprint or str) – The LLM blueprint to test, either LLMBlueprint or the LLM blueprint ID.

Returns:

llm_test_result – The created LLM test result.

Return type:

LLMTestResult

classmethod get(llm_test_result)

Retrieve a single LLM test result.

Parameters:: llm_test_result (LLMTestResult or str) – The LLM test result to retrieve, specified by either LLM test result or test ID.
Returns:: llm_test_result – The requested LLM test result.
Return type:: LLMTestResult

classmethod list(llm_test_configuration=None, llm_blueprint=None)

List all LLM test results available to the user. If the LLM test configuration or LLM blueprint is specified, results are restricted to only those LLM test results associated with the LLM test configuration or LLM blueprint.

Parameters:

llm_test_configuration (Optional[Union[LLMTestConfiguration, str]]) – The returned LLM test results are filtered to those associated with a specific LLM test configuration, if specified.
llm_blueprint (Optional[Union[LLMBlueprint, str]]) – The returned LLM test results, filtered by those associated with a specific LLM blueprint, if specified.

Returns:

llm_test_results – Returns a list of LLM test results.

Return type:

List[LLMTestResult]

delete()

Delete a single LLM test result.

Return type:: None

class datarobot.models.genai.llm_test_configuration.DatasetEvaluation

Metadata for a DataRobot GenAI dataset evaluation.

Variables:

evaluation_name (str) – The name of the evaluation.
evaluation_dataset_configuration_id (str or None, optional) – The ID of the evaluation dataset configuration for custom datasets.
evaluation_dataset_name (str) – The name of the evaluation dataset.
ootb_dataset (OOTBDataset or None, optional) – Out-of-the-box dataset.
insight_configuration (InsightsConfiguration) – The insight to calculate for this dataset.
insight_grading_criteria (InsightGradingCriteria) – The criteria to use for grading the results.
max_num_prompts (int) – The maximum number of prompts to use for the evaluation.
prompt_sampling_strategy (PromptSamplingStrategy) – The prompt sampling strategy for the dataset evaluation.

class datarobot.models.genai.llm_test_result.InsightEvaluationResult

Metadata for a DataRobot GenAI insight evaluation result.

Variables:

id (str) – The ID of the insight evaluation result.
llm_test_result_id (str) – The ID of the LLM test result associated with this insight evaluation result.
evaluation_dataset_configuration_id (str) – The ID of the evaluation dataset configuration.
evaluation_dataset_name (str) – The name of the evaluation dataset.
metric_name (str) – The name of the metric.
chat_id (str) – The ID of the chat containing the prompts and responses.
chat_name (str) – The name of the chat containing the prompts and responses.
aggregation_type (AggregationType) – The type of aggregation used for the metric results.
grading_result (GradingResult) – The overall grade for the LLM test.
execution_status (str) – The execution status of the LLM test.
evaluation_name (str) – The name of the evaluation.
insight_grading_criteria (InsightGradingCriteria) – The criteria to grade the results.
last_update_date (str) – The date the result was most recently updated.
aggregation_value (float | List[Dict[str, float]] | None) – The aggregated metric result.

class datarobot.models.genai.llm_test_configuration.OOTBDatasetDict

class datarobot.models.genai.llm_test_configuration.DatasetEvaluationRequestDict

class datarobot.models.genai.llm_test_configuration.DatasetEvaluationDict

class datarobot.models.genai.nemo_configuration.NemoConfiguration

Configuration for the Nemo Pipeline.

Variables:

prompt_pipeline_metric_name (Optional[str]) – The name of the metric for the prompt pipeline.
prompt_pipeline_files (NemoFileContentsResponse, optional) – The files used in the prompt pipeline.
prompt_llm_configuration (NemoLLMConfiguration, optional) – The LLM configuration for the prompt pipeline.
prompt_moderation_configuration (ModerationConfigurationWithoutID, optional) – The moderation configuration for the prompt pipeline.
prompt_pipeline_template_id (Optional[str]) – The ID of the prompt pipeline template. This parameter defines the actions.py file.
response_pipeline_metric_name (Optional[str]) – The name of the metric for the response pipeline.
response_pipeline_files (NemoFileContentsResponse, optional) – The files used in the response pipeline.
response_llm_configuration (NemoLLMConfiguration, optional) – The LLM configuration for the response pipeline.
response_moderation_configuration (ModerationConfigurationWithoutID, optional) – The moderation configuration for the response pipeline.
response_pipeline_template_id (Optional[str]) – The ID of the response pipeline template. This parameter defines the actions.py file.
blocked_terms_file_contents (str) – The contents of the blocked terms file. This is shared between the prompt and response pipelines.

classmethod get(playground)

Get the Nemo configuration for a playground.

Parameters:: playground (str or Playground) – The playground to get the configuration for
Returns:: The Nemo configuration for the playground.
Return type:: NemoConfiguration

classmethod upsert(playground, blocked_terms_file_contents, prompt_pipeline_metric_name=None, prompt_pipeline_files=None, prompt_llm_configuration=None, prompt_moderation_configuration=None, prompt_pipeline_template_id=None, response_pipeline_metric_name=None, response_pipeline_files=None, response_llm_configuration=None, response_moderation_configuration=None, response_pipeline_template_id=None)

Create or update the nemo configuration for a playground.

Parameters:

playground (str or Playground) – The playground for the configuration
blocked_terms_file_contents (str) – The contents of the blocked terms file.
prompt_pipeline_metric_name (Optional[str]) – The name of the metric for the prompt pipeline.
prompt_pipeline_files (NemoFileContents, optional) – The files used in the prompt pipeline.
prompt_llm_configuration (NemoLLMConfiguration, optional) – The LLM configuration for the prompt pipeline.
prompt_moderation_configuration (ModerationConfigurationWithoutID, optional) – The moderation configuration for the prompt pipeline.
prompt_pipeline_template_id (Optional[str]) – The ID of the prompt pipeline template, this will define the action.py file.
response_pipeline_metric_name (Optional[str]) – The name of the metric for the response pipeline.
response_pipeline_files (NemoFileContents, optional) – The files used in the response pipeline.
response_llm_configuration (NemoLLMConfiguration, optional) – The LLM configuration for the response pipeline.
response_moderation_configuration (ModerationConfigurationWithoutID, optional) – The moderation configuration for the response pipeline.
response_pipeline_template_id (Optional[str]) – The ID of the response pipeline template, this will define the action.py file.

Returns:

The Nemo configuration for the playground.

Return type:

NemoConfiguration

class datarobot.models.genai.llm_test_configuration.OOTBDataset

Metadata for a DataRobot GenAI out-of-the-box LLM compliance test dataset.

Variables:

dataset_name (str) – The name of the dataset.
prompt_column_name (str) – The name of the prompt column.
response_column_name (str or None, optional) – The name of the response column, if any.
dataset_url (str or None, optional) – The URL of the dataset.
rows_count (int) – The number of rows in the dataset.
warning (str or None, optional) – A warning message regarding the contents of the dataset, if any.

classmethod list()

List all out-of-the-box datasets available to the user.

Returns:: ootb_datasets – Returns a list of out-of-the-box datasets.
Return type:: list[OOTBDataset]

class datarobot.models.genai.llm_test_configuration.NonOOTBDataset

Metadata for a DataRobot GenAI non out-of-the-box (OOTB) LLM compliance test dataset.

classmethod list(use_case=None)

List all non out-of-the-box datasets available to the user.

Returns:: non_ootb_datasets – Returns a list of non out-of-the-box datasets.
Return type:: list[NonOOTBDataset]

class datarobot.models.genai.prompt_trace.TraceMetadata

Trace metadata contains information about all the users and chats that are relevant to this playground.

Variables:: users (list[dict]) – The users who submitted the prompt.

classmethod get(playground)

Get trace metadata for a playground.

Parameters:: playground (str) – The ID of the playground to get trace metadata for.
Returns:: trace_metadata – The trace metadata for the playground.
Return type:: TraceMetadata

class datarobot.models.genai.metric_insights.MetricInsights

Metric insights for playground.

classmethod list(playground)

Get metric insights for playground.

Parameters:: playground (str or Playground) – Playground to get the supported metrics from.
Returns:: insights – Metric insights for playground.
Return type:: list[InsightsConfiguration]

classmethod copy_to_playground(source_playground, target_playground, add_to_existing=True, with_evaluation_datasets=False)

Copy metric insights to from one playground to another.

Parameters:

source_playground (str or Playground) – Playground to copy metric insights from.
target_playground (str or Playground) – Playground to copy metric insights to.
add_to_existing (Optional[bool]) – Add metric insights to existing ones in the target playground, by default True.
with_evaluation_datasets (Optional[bool]) – Copy evaluation datasets from the source playground.

Return type:

None

class datarobot.models.genai.ootb_metric_configuration.PlaygroundOOTBMetricConfiguration

OOTB metric configurations for a playground.

Variables:: ootb_metric_configurations ((List[OOTBMetricConfigurationResponse]): The list of the OOTB metric configurations.)

classmethod get(playground_id)

Get OOTB metric configurations for the playground.

Return type:: PlaygroundOOTBMetricConfiguration

classmethod create(playground_id, ootb_metric_configurations)

Create a new OOTB metric configurations.

Return type:: PlaygroundOOTBMetricConfiguration