Generative AI 

class datarobot.models.genai.comparison_chat.ComparisonChat(id, name, playground_id, creation_date, creation_user_id)

Metadata for a DataRobot GenAI comparison chat.

Attributes:

idstr: The comparison chat ID.
namestr: The comparison chat name.
playground_idstr: The ID of the playground associated with the comparison chat.
creation_datestr: The date when the comparison chat was created.
creation_user_idstr: The ID of the creating user.

classmethod create(name, playground)

Creates a new comparison chat.

Parameters:

namestr: The comparison chat name.
playgroundPlayground or str: The playground associated with the created comparison chat, either Playground or playground ID.

Returns:

comparison_chatComparisonChat: The created comparison chat.

Return type:

ComparisonChat

classmethod get(comparison_chat)

Retrieve a single comparison chat.

Parameters:

comparison_chatComparisonChat or str: The comparison chat you want to retrieve. Accepts ComparisonChat or comparison chat ID.

Returns:

comparison_chatComparisonChat: The requested comparison chat.

Return type:

ComparisonChat

classmethod list(playground=None, sort=None)

List all comparison chats available to the user. If the playground is specified, results are restricted to only those comparison chats associated with the playground.

Parameters:

playgroundOptional[Union[Playground, str]], optional: Returns only those comparison chats associated with a particular playground, specified by either the Playground or the playground ID.
sortstr, optional: The property to sort comparison chats by. Prefix the attribute name with a dash ( - ) to sort responses in descending order, (for example, ‘-name’). Supported options are listed in ListComparisonChatsSortQueryParams, but the values can differ depending on platform version. The default sort parameter is None, which results in comparison chats returning in order of creation time, descending.

Returns:

comparison_chatslist[ComparisonChat]: Returns a list of comparison chats.

Return type:

List[ComparisonChat]

delete()

Delete the single comparison chat.

Return type:: None

update(name)

Update the comparison chat.

Parameters:

namestr: The new name for the comparison chat.

Returns:

comparison_chatComparisonChat: The updated comparison chat.

Return type:

ComparisonChat

class datarobot.models.genai.comparison_prompt.ComparisonPrompt(id, text, results, creation_date, creation_user_id, comparison_chat_id=None, metadata_filter=None)

Metadata for a DataRobot GenAI comparison prompt.

Attributes:

idstr

Comparison prompt ID.

textstr

The prompt text.

resultslist[ComparisonPromptResult]

The list of results for individual LLM blueprints that are part of the comparison prompt.

creation_datestr

The date when the playground was created.

creation_user_idstr

ID of the creating user.

comparison_chat_idstr

The ID of the comparison chat this comparison prompt is associated with.

metadata_filter: Optional[Dict[str, Any] | None]

The metadata filter to apply to the vector database.

Supports: - None or empty dict (no filters): Considers all documents - Multiple field filters (implicit AND): {“a”: 1, “b”: “b”} - Comparison operators: {“field”: {“$gt”: 5}} - Logical operators: {“$and”: […], “$or”: […]} - Nested combinations of the above

Comparison operators: - $eq: equal to (string, int, float, bool) - $ne: not equal to (string, int, float, bool) - $gt: greater than (int, float) - $gte: greater than or equal to (int, float) - $lt: less than (int, float) - $lte: less than or equal to (int, float) - $in: a value is in list (string, int, float, bool) - $nin: a value is not in list (string, int, float, bool) - $contains: a string contains a value (string) - $not_contains: a string does not contain a value (string)

update(additional_llm_blueprints=None, wait_for_completion=False, feedback_result=None, **kwargs)

Update the comparison prompt.

Parameters:

additional_llm_blueprintslist[LLMBlueprint or str]: The additional LLM blueprints you want to submit the comparison prompt.

Returns:

comparison_promptComparisonPrompt: The updated comparison prompt.

Return type:

ComparisonPrompt

classmethod create(llm_blueprints, text, comparison_chat=None, wait_for_completion=False)

Create a new ComparisonPrompt. This submits the prompt text to the LLM blueprints that are specified.

Parameters:

llm_blueprintslist[LLMBlueprint or str]: The LLM blueprints associated with the created comparison prompt. Accepts LLM blueprints or IDs.
textstr: The prompt text.
comparison_chat: Optional[ComparisonChat or str], optional: The comparison chat to add the comparison prompt to. Accepts ComparisonChat or comparison chat ID.
wait_for_completionbool: If set to True code will wait for the chat prompt job to complete before returning the result (up to 10 minutes, raising timeout error after that). Otherwise, you can check current status by using ChatPrompt.get with returned ID.

Returns:

comparison_promptComparisonPrompt: The created comparison prompt.

Return type:

ComparisonPrompt

classmethod get(comparison_prompt)

Retrieve a single comparison prompt.

Parameters:

comparison_promptstr: The comparison prompt you want to retrieve. Accepts entity or ID.

Returns:

comparison_promptComparisonPrompt: The requested comparison prompt.

Return type:

ComparisonPrompt

classmethod list(llm_blueprints=None, comparison_chat=None)

List all comparison prompts available to the user that include the specified LLM blueprints or from the specified comparison chat.

Parameters:

llm_blueprintsOptional[List[Union[LLMBlueprint, str]]], optional: The returned comparison prompts are only those associated with the specified LLM blueprints. Accepts either LLMBlueprint or LLM blueprint ID.
comparison_chatOptional[Union[ComparisonChat, str]], optional: The returned comparison prompts are only those associated with the specified comparison chat. Accepts either ComparisonChat or comparison chat ID.

Returns:

comparison_promptslist[ComparisonPrompt]: A list of comparison prompts available to the user that use the specified LLM blueprints.

Return type:

List[ComparisonPrompt]

delete()

Delete the single comparison prompt.

Return type:: None

class datarobot.models.genai.custom_model_validation.CustomModelValidation(id, prompt_column_name, target_column_name, deployment_id, model_id, validation_status, deployment_access_data, tenant_id, name, creation_date, user_id, error_message, deployment_name, user_name, use_case_id, prediction_timeout)

Validation record checking the ability of the deployment to serve as a custom model LLM or vector database.

Attributes:

prompt_column_namestr: The column name that the deployed model expects as the input.
target_column_namestr: The target name that the deployed model will output.
deployment_idstr: ID of the deployment.
model_idstr: ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].
validation_statusstr: Can be TESTING, FAILED, or PASSED. Only PASSED is allowed for use.
deployment_access_datadict, optional: Data that will be used for accessing deployment prediction server. Only available for deployments that pass validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - First of two auth headers for the prediction server. - authorization_header - Second of two auth headers for the prediction server. - input_type - Either JSON or CSV - input type model expects. - model_type - Target type of deployed custom model.
tenant_idstr: Creating user’s tenant ID.
error_messageOptional[str]: Additional information for errored validation.
deployment_nameOptional[str]: The name of the deployment that is validated.
user_nameOptional[str]: The name of the user
use_case_idOptional[str]: The ID of the Use Case associated with the validation.
prediction_timeout: int: The timeout, in seconds, for the prediction API used in this custom model validation.

classmethod get(validation_id)

Get the validation record by id.

Parameters:

validation_idUnion[CustomModelValidation, str]: The CustomModelValidation to retrieve, either CustomModelValidation or validation ID.

Returns:

CustomModelValidation

Return type:

classmethod get_by_values(prompt_column_name, target_column_name, deployment_id, model_id)

Get the validation record by field values.

Parameters:

prompt_column_namestr: The column name the deployed model expect as the input.
target_column_namestr: The target name deployed model will output.
deployment_idstr: ID of the deployment.
model_idstr: ID of the underlying deployment model.

Returns:

CustomModelValidation

Return type:

classmethod list(prompt_column_name=None, target_column_name=None, deployment=None, model=None, use_cases=None, playground=None, completed_only=False, search=None, sort=None)

List the validation records by field values.

Parameters:

prompt_column_nameOptional[str], optional: The column name the deployed model expects as the input.
target_column_nameOptional[str], optional: The target name that the deployed model will output.
deploymentOptional[Union[Deployment, str]], optional: The returned validations are filtered to those associated with a specific deployment if specified, either Deployment or deployment ID.
model_idOptional[Union[Model, str]], optional: The returned validations are filtered to those associated with a specific model if specified, either Model or model ID.
use_casesOptional[list[Union[UseCase, str]]], optional: The returned validations are filtered to those associated with specific Use Cases if specified, either UseCase or Use Case IDs.
playground_idOptional[Union[Playground, str]], optional: The returned validations are filtered to those used in a specific playground if specified, either Playground or playground ID.
completed_onlybool, optional: Whether to retrieve only completed validations.
searchOptional[str], optional: String for filtering validations. Validations that contain the string in name will be returned.
sortOptional[str], optional: Property to sort validations by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-name’. Currently supported options are listed in ListCustomModelValidationsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in validations being returned in order of creation time descending.

Returns:

List[CustomModelValidation]

Return type:

List[CustomModelValidation]

classmethod create(prompt_column_name, target_column_name, deployment_id, model=None, use_case=None, name=None, wait_for_completion=False, prediction_timeout=None)

Start the validation of deployment to serve as a vector database or LLM.

Parameters:

prompt_column_namestr: The column name the deployed model expect as the input.
target_column_namestr: The target name that the deployed model will output.
deployment_idUnion[Deployment, str]: The deployment to validate, either Deployment or deployment ID.
modelOptional[Union[Model, str]], optional: The specific model within the deployment, either Model or model ID. If not specified, the underlying model ID will be derived from the deployment info automatically.
use_caseOptional[Union[UseCase, str]], optional: The Use Case to link the validation to, either UseCase or Use Case ID.
nameOptional[str], optional: The name of the validation.
wait_for_completionbool: If set to True code will wait for the validation job to complete before returning the result (up to 10 minutes, raising timeout error after that). Otherwise, you can check current validation status by using CustomModelValidation.get with returned ID.
prediction_timeoutOptional[int], optional: The timeout, in seconds, for the prediction API used in this custom model validation.

Returns:

CustomModelValidation

Return type:

classmethod revalidate(validation_id)

Revalidate an unlinked custom model vector database or LLM. This method is useful when a deployment used as vector database or LLM is accidentally replaced with another model that stopped complying with the vector database or LLM requirements. Replace the model back and call this method instead of creating a new custom model validation from scratch. Another use case for this is when the API token used to create a validation record got revoked and no longer can be used by vector database / LLM to call custom model deployment. Calling revalidate will update the validation record with the token currently in use.

Parameters:

validation_idstr: The ID of the CustomModelValidation for revalidation.

Returns:

CustomModelValidation

Return type:

update(name=None, prompt_column_name=None, target_column_name=None, deployment=None, model=None, prediction_timeout=None)

Update a custom model validation.

Parameters:

nameOptional[str], optional: The new name of the custom model validation.
prompt_column_nameOptional[str], optional: The new name of the prompt column.
target_column_nameOptional[str], optional: The new name of the target column.
deploymentOptional[Union[Deployment, str]], optional: The new deployment to validate.
modelOptional[Union[Model, str]], optional: The new model within the deployment to validate.
prediction_timeoutOptional[int], optional: The new timeout, in seconds, for the prediction API used in this custom model validation.

Returns:

CustomModelValidation

Return type:

delete()

Delete the custom model validation.

Return type:: None

class datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation(id, prompt_column_name, target_column_name, deployment_id, model_id, validation_status, deployment_access_data, tenant_id, name, creation_date, user_id, error_message, deployment_name, user_name, use_case_id, prediction_timeout)

Validation record checking the ability of the deployment to serve as a custom model LLM.

Attributes:

prompt_column_namestr: The column name the deployed model expect as the input.
target_column_namestr: The target name that the deployed model will output.
deployment_idstr: ID of the deployment.
model_idstr: ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].
validation_statusstr: Can be TESTING, FAILED, or PASSED. Only PASSED is allowed for use.
deployment_access_datadict, optional: Data that will be used for accessing deployment prediction server. Only available for deployments that passed validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - first of 2 auth headers for the prediction server. - authorization_header - second of 2 auth headers for the prediction server. - input_type - Either JSON or CSV - the input type that the model expects. - model_type - Target type of the deployed custom model.
tenant_idstr: Creating user’s tenant ID.
error_messageOptional[str]: Additional information for errored validation.
deployment_nameOptional[str]: The name of the deployment that is validated.
user_nameOptional[str]: The name of the user
use_case_idOptional[str]: The ID of the Use Case associated with the validation.
prediction_timeout: int: The timeout in seconds for the prediction API used in this custom model validation.

class datarobot.models.genai.vector_database.CustomModelVectorDatabaseValidation(id, prompt_column_name, target_column_name, deployment_id, model_id, validation_status, deployment_access_data, tenant_id, name, creation_date, user_id, error_message, deployment_name, user_name, use_case_id, prediction_timeout)

Validation record checking the ability of the deployment to serve as a vector database.

Attributes:

prompt_column_namestr: The column name the deployed model expect as the input.
target_column_namestr: The target name deployed model will output.
deployment_idstr: ID of the deployment.
model_idstr: ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].
validation_statusstr: Can be TESTING, FAILED and PASSED. Only PASSED allowed for use.
deployment_access_datadict, optional: Data that will be used for accessing deployment prediction server. Only available for deployments that passed validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - first of 2 auth headers for prediction server. - authorization_header - second of 2 auth headers for prediction server. - input_type - Either JSON or CSV - input type model expects. - model_type - Target type of deployed custom model.
tenant_idstr: Creating user’s tenant ID.
error_messageOptional[str]: Additional information for errored validation.
deployment_nameOptional[str]: The name of the deployment that is validated.
user_nameOptional[str]: The name of the user
use_case_idOptional[str]: The ID of the use case associated with the validation.

class datarobot.models.genai.llm_blueprint.LLMBlueprint(id, name, description, is_saved, is_starred, playground_id, creation_date, creation_user_id, creation_user_name, last_update_date, last_update_user_id, prompt_type, llm_id=None, llm_name=None, llm_settings=None, vector_database_id=None, vector_database_settings=None, vector_database_name=None, vector_database_status=None, vector_database_error_message=None, vector_database_error_resolution=None, custom_model_llm_validation_status=None, custom_model_llm_error_message=None, custom_model_llm_error_resolution=None)

Metadata for a DataRobot GenAI LLM blueprint.

Attributes:

idstr: The LLM blueprint ID.
namestr: The LLM blueprint name.
descriptionstr: A description of the LLM blueprint.
is_savedbool: Whether the LLM blueprint is saved (meaning the settings are locked and blueprint is eligible for use with ComparisonPrompts).
is_starredbool: Whether the LLM blueprint is starred.
playground_idstr: The ID of the playground associated with the LLM blueprint.
llm_idstr or None: The ID of the LLM type. If not None, this must be one of the IDs returned by LLMDefinition.list for this user.
llm_namestr or None: The name of the LLM.
llm_settingsdict or None: The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the external model LLM validation. - external_llm_context_size - The external LLM’s context size, in tokens, for external model LLM blueprints.
creation_datestr: The date the playground was created.
creation_user_idstr: The ID of the user creating the playground.
creation_user_namestr: The name of the user creating the playground.
last_update_datestr: The date the playground was last updated.
last_update_user_idstr: The ID of the user who most recently updated the playground.
prompt_typePromptType: The prompting strategy for the LLM blueprint. Currently supported options are listed in PromptType.
vector_database_idstr or None: The ID of the vector database, if any, associated with the LLM blueprint.
vector_database_settingsVectorDatabaseSettings or None: The settings for the vector database, if any, associated with the LLM blueprint.
vector_database_namestr or None: The name of the vector database associated with the LLM blueprint, if any.
vector_database_statusstr or None: The status of the vector database, if any, associated with the LLM blueprint.
vector_database_error_messagestr or None: The error message for the vector database, if any, associated with the LLM blueprint.
vector_database_error_resolutionstr or None: The resolution for the vector database error, if any, associated with the LLM blueprint.
custom_model_llm_validation_statusstr or None: The status of the custom model LLM validation if the llm_id is ‘custom-model’.
custom_model_llm_error_messagestr or None: The error message for the custom model LLM, if any.
custom_model_llm_error_resolutionstr or None: The resolution for the custom model LLM error, if any.

classmethod create(playground, name, prompt_type=PromptType.CHAT_HISTORY_AWARE, description='', llm=None, llm_settings=None, vector_database=None, vector_database_settings=None)

Create a new LLM blueprint.

Parameters:

playgroundPlayground or str: The playground associated with the created LLM blueprint. Accepts playground or playground ID.
namestr: The LLM blueprint name.
prompt_typePromptType, optional: Prompting type of the LLM blueprint, by default PromptType.CHAT_HISTORY_AWARE.
descriptionstr, optional: An optional description for the LLM blueprint, otherwise null.
llmLLMDefinition, str, or None, optional: The LLM to use for the blueprint, either LLMDefinition or LLM ID.
llm_settingsdict or None: The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.
vector_database: VectorDatabase, str, or None, optional: The vector database to use with this LLM blueprint, either VectorDatabase or vector database ID.
vector_database_settings: VectorDatabaseSettings or None, optional: The settings for the vector database, if any.

Returns:

llm_blueprintLLMBlueprint: The created LLM blueprint.

Return type:

classmethod create_from_llm_blueprint(llm_blueprint, name, description='')

Create a new LLM blueprint from an existing LLM blueprint.

Parameters:

llm_blueprintLLMBlueprint or str: The LLM blueprint to use to create the new LLM blueprint. Accepts LLM blueprint or LLM blueprint ID.
namestr: LLM blueprint name.
descriptionstr, optional: Description of the LLM blueprint, by default “”.

Returns:

llm_blueprintLLMBlueprint: The created LLM blueprint.

Return type:

classmethod get(llm_blueprint_id)

Retrieve a single LLM blueprint.

Parameters:

llm_blueprint_idstr: The ID of the LLM blueprint you want to retrieve.

Returns:

llm_blueprintLLMBlueprint: The requested LLM blueprint.

Return type:

classmethod list(playground=None, llms=None, vector_databases=None, is_saved=None, is_starred=None, sort=None)

Lists all LLM blueprints available to the user. If the playground is specified, then the results are restricted to the LLM blueprints associated with the playground. If the LLMs are specified, then the results are restricted to the LLM blueprints using those LLM types. If vector_databases are specified, then the results are restricted to the LLM blueprints using those vector databases.

Parameters:

playgroundOptional[Union[Playground, str]], optional: The returned LLM blueprints are filtered to those associated with a specific playground if it is specified. Accepts either the entity or the ID.
llmsOptional[list[Union[LLMDefinition, str]]], optional: The returned LLM blueprints are filtered to those associated with the LLM types specified. Accepts either the entity or the ID.
vector_databasesOptional[list[Union[VectorDatabase, str]]], optional: The returned LLM blueprints are filtered to those associated with the vector databases specified. Accepts either the entity or the ID.
is_saved: Optional[bool], optional: The returned LLM blueprints are filtered to those matching is_saved.
is_starred: Optional[bool], optional: The returned LLM blueprints are filtered to those matching is_starred.
sortstr, optional: Property to sort LLM blueprints by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListLLMBlueprintsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in LLM blueprints being returned in order of creation time descending.

Returns:

playgroundslist[Playground]: A list of playgrounds available to the user.

Return type:

List[LLMBlueprint]

update(name=None, description=None, llm=None, llm_settings=None, vector_database=None, vector_database_settings=None, is_saved=None, is_starred=None, prompt_type=None, remove_vector_database=False)

Update the LLM blueprint.

Parameters:

namestr or None, optional: The new name for the LLM blueprint.
description: str or None, optional: The new description for the LLM blueprint.
llm: Optional[Union[LLMDefinition, str]], optional: The new LLM type for the LLM blueprint.
llm_settings: Optional[dict], optional: The new LLM settings for the LLM blueprint. These must match the LLMSettings returned from the LLMDefinition.list method for the LLM type used for this LLM blueprint but this typically has dict fields: - system_prompt - The system prompt that tells the LLM how to behave. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Whether the model considers next tokens with top_p probability mass. Or - system_prompt - The system prompt that tells the LLM how to behave. - validation_id - The ID of the custom model LLM validation for custom model LLM blueprints.
vector_database: Optional[Union[VectorDatabase, str]], optional: The new vector database for the LLM blueprint.
vector_database_settings: Optional[VectorDatabaseSettings], optional: The new vector database settings for the LLM blueprint.
is_saved: Optional[bool], optional: The new is_saved attribute for the LLM blueprint (meaning the settings are locked and blueprint is eligible for use with ComparisonPrompts).
is_starred: Optional[bool], optional: The new setting for whether the LLM blueprint is starred.
prompt_typePromptType, optional: The new prompting type of the LLM blueprint.
remove_vector_database: Optional[bool], optional: Whether to remove the vector database from the LLM blueprint.

Returns:

llm_blueprintLLMBlueprint: The updated LLM blueprint.

Return type:

VectorDatabaseDatasetExportJob

delete()

Delete the single LLM blueprint.

Return type:: None

register_custom_model(prompt_column_name=None, target_column_name=None, llm_test_configuration_ids=None)

Create a new CustomModelVersion. This registers a custom model from the LLM blueprint.

Parameters:

prompt_column_namestr, optional: The column name of the prompt text.
target_column_namestr, optional: The column name of the response text.
llm_test_configuration_idsList[str], optional: The IDs of the LLM test configurations to execute.

Returns:

custom_modelCustomModelVersion: The registered custom model.

Return type:

CustomModelVersion

class datarobot.models.genai.llm.LLMDefinition(id, name, description, vendor, license, supported_languages, settings, context_size=None)

Metadata for a DataRobot GenAI LLM.

Attributes:

idstr: Language model type ID.
namestr: Language model name.
descriptionstr: Description of the language model.
vendorstr: Name of the vendor for this model.
licensestr: License for this model.
supported_languagesstr: Languages supported by this model.
settingslist of LLMSettingDefinition: Settings for this model
context_sizeint: The context size for this model

classmethod list(use_case=None, as_dict=True)

List all large language models (LLMs) available to the user.

Parameters:

use_caseOptional[UseCase or str], optional: The returned LLMs, including external LLMs, available for the specified Use Case. Accepts either the entity or the Use CaseID.

Returns:

llmslist[LLMDefinition] or list[LLMDefinitionDict]: A list of large language models (LLMs) available to the user.

Return type:

Union[List[LLMDefinition], List[LLMDefinitionDict]]

class datarobot.models.genai.llm.LLMDefinitionDict(*args, **kwargs)

class datarobot.models.genai.playground.Playground(id, name, description, use_case_id, creation_date, creation_user_id, last_update_date, last_update_user_id, saved_llm_blueprints_count, llm_blueprints_count, user_name)

Metadata for a DataRobot GenAI playground.

Attributes:

idstr: Playground ID.
namestr: Playground name.
descriptionstr: Description of the playground.
use_case_idstr: Linked use case ID.
creation_datestr: The date when the playground was created.
creation_user_idstr: ID of the creating user.
last_update_datestr: Date when the playground was most recently updated.
last_update_user_idstr: ID of the user who most recently updated the playground.
saved_llm_blueprints_countint: Number of saved LLM blueprints in the playground.
llm_blueprints_countint: Number of LLM blueprints in the playground.
user_namestr: The name of the user who created the playground.

classmethod create(name, description='', use_case=None, copy_insights=None)

Create a new playground.

Parameters:

namestr: Playground name.
descriptionstr, optional: Description of the playground, by default “”.
use_caseOptional[Union[UseCase, str]], optional: Use case to link to the created playground.
copy_insights: CopyInsightsRequest, optional: If present, copies insights from the source playground to the created playground.

Returns:

playgroundPlayground: The created playground.

Return type:

Playground

classmethod get(playground_id)

Retrieve a single playground.

Parameters:

playground_idstr: The ID of the playground you want to retrieve.

Returns:

playgroundPlayground: The requested playground.

Return type:

Playground

classmethod list(use_case=None, search=None, sort=None)

List all playgrounds available to the user. If the use_case is specified or can be inferred from the Context then the results are restricted to the playgrounds associated with the UseCase.

Parameters:

use_caseOptional[UseCaseLike], optional: The returned playgrounds are filtered to those associated with a specific Use Case or Cases if specified or can be inferred from the Context. Accepts either the entity or the ID.
searchstr, optional: String for filtering playgrounds. Playgrounds that contain the string in name will be returned. If not specified, all playgrounds will be returned.
sortstr, optional: Property to sort playgrounds by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListPlaygroundsSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in playgrounds being returned in order of creation time descending.

Returns:

playgroundslist[Playground]: A list of playgrounds available to the user.

Return type:

List[Playground]

update(name=None, description=None)

Update the playground.

Parameters:

namestr: The new name for the playground.
description: str: The new description for the playground.

Returns:

playgroundPlayground: The updated playground.

Return type:

Playground

delete()

Delete the playground.

Return type:: None

class datarobot.enums.PromptType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None): Supported LLM blueprint prompting types.

class datarobot.models.genai.vector_database.SupportedEmbeddings(embedding_models, default_embedding_model, custom_model_embedding_validations)

All supported embedding models including the recommended default model.

Attributes:

embedding_modelslist[EmbeddingModel]: All supported embedding models.
default_embedding_modelstr: Name of the default recommended text embedding model. Currently supported options are listed in VectorDatabaseEmbeddingModel but the values can differ with different platform versions.
custom_model_embedding_validationsList[str]: External embedding models that have been validated

class datarobot.models.genai.vector_database.SupportedTextChunkings(text_chunking_configs)

Supported text chunking configurations which includes a set of recommended chunking parameters for each supported embedding model.

Attributes:

text_chunking_configs: All supported text chunking configurations.

class datarobot.models.genai.user_limits.UserLimits(counter)

Counts for user limits for LLM APIs and vector databases.

classmethod get_vector_database_count()

Get the count of vector databases for the user.

Return type:: APIObject

classmethod get_llm_requests_count()

Get the count of LLMs requests made by the user.

Return type:: APIObject

class datarobot.models.genai.vector_database.VectorDatabase(id, name, size, use_case_id, dataset_id, embedding_model, chunking_method, chunk_size, chunk_overlap_percentage, chunks_count, custom_chunking, separators, creation_date, creation_user_id, organization_id, tenant_id, last_update_date, execution_status, playgrounds_count, dataset_name, user_name, source, validation_id, error_message, is_separator_regex, embedding_validation_id, parent_id, family_id, metada_columns, added_dataset_ids, added_dataset_names, version)

Metadata for a DataRobot vector database accessible to the user.

Attributes:

idstr: Vector database ID.
namestr: Vector database name.
sizeint: Size of the vector database assets in bytes.
use_case_idstr: Linked use case ID.
dataset_idstr: ID of the dataset used for creation.
embedding_modelstr: Name of the text embedding model. Currently supported options are listed in VectorDatabaseEmbeddingModel but the values can differ with different platform versions.
chunking_methodstr or None: Name of the method to split dataset documents. Currently supported options are listed in VectorDatabaseChunkingMethod but the values can differ with different platform versions.
chunk_sizeint or None: Size of each text chunk in number of tokens.
chunk_overlap_percentageint or None: Overlap percentage between chunks.
chunks_countint: Total number of text chunks.
custom_chunkingbool: Determines if the chunking is custom. With custom chunking, dataset rows are not split into chunks automatically; instead, the user provides the chunks.
separatorslist[string] or None: Separators for document splitting.
creation_datestr: Date when the database was created.
creation_user_idstr: ID of the creating user.
organization_idstr: Creating user’s organization ID.
tenant_idstr: Creating user’s tenant ID.
last_update_datestr: Last update date for the database.
execution_statusstr: Database execution status. Currently supported options are listed in VectorDatabaseExecutionStatus but the values can differ with different platform versions.
playgrounds_countint: Number of using playgrounds.
dataset_namestr: Name of the used dataset.
user_namestr: Name of the creating user.
sourcestr: Source of the vector database. Currently supported options are listed in VectorDatabaseSource but the values can differ with different platform versions.
validation_idOptional[str]: ID of custom model vector database validation. Only filled for external vector databases.
error_messageOptional[str]: Additional information for errored vector database.
embedding_validation_idOptional[str]: ID of the custom embedding validation, if any.
is_separator_regexbool: Whether the separators should be treated as regular expressions.

classmethod get_supported_embeddings(dataset_id=None, use_case=None)

Get all supported and the recommended embedding models.

Parameters:

dataset_idstr, optional: ID of a dataset for which the recommended model is returned based on the detected language of that dataset.
use_caseOptional[UseCase, str]: May be Use Case ID or the Use Case entity.

Returns:

supported_embeddingsSupportedEmbeddings: The supported embedding models.

Return type:

SupportedEmbeddings

submit_export_dataset_job()

Submit the vector database dataset export job.

Returns:

resultVectorDatabaseDatasetExportJob: The result of the vector database dataset export job containing the exported dataset id.

Return type:

classmethod get_supported_retrieval_settings()

Get supported retrieval settings.

Returns:

supported_retrieval_settingsSupportedRetrievalSettings: The supported retriever settings.

Return type:

SupportedRetrievalSettings

classmethod create(dataset_id, chunking_parameters=None, use_case=None, name=None, parent_vector_database_id=None, update_llm_blueprints=None, update_deployments=None)

Create a new vector database.

Parameters:

dataset_idstr: ID of the dataset used for creation.
chunking_parametersChunkingParameters: Parameters defining how documents are split and embedded.
use_caseOptional[Union[UseCase, str]], optional: Use case to link to the created vector database.
namestr, optional: Vector database name, by default None which leads to the default name ‘Vector Database for <dataset name>’.
parent_vector_database_idstr, optional: ID of the parent vector database to base the update on.
update_llm_blueprintsbool, optional: Whether to update LLM blueprints related to the parent vector database.
update_deploymentsbool, optional: Whether to update deployments related to the parent vector database.

Returns:

vector databaseVectorDatabase: The created vector database with execution status ‘new’.

Return type:

classmethod create_from_custom_model(name, use_case=None, validation_id=None, prompt_column_name=None, target_column_name=None, deployment_id=None, model_id=None)

Create a new vector database from validated custom model deployment.

Parameters:

namestr: Vector database name.
use_caseOptional[Union[UseCase, str]], optional: Use case to link to the created vector database.
validation_idstr, optional: ID of CustomModelVectorDatabaseValidation for the deployment. Alternatively, you can specify ALL the following fields.
prompt_column_namestr, optional: The column name the deployed model expect as the input.
target_column_namestr, optional: The target name deployed model will output.
deployment_idstr, optional: ID of the deployment.
model_idstr, optional: ID of the underlying deployment model. Can be found from the API as Deployment.model[“id”].

Returns:

vector databaseVectorDatabase: The created vector database.

Return type:

classmethod get(vector_database_id)

Retrieve a single vector database.

Parameters:

vector_database_idstr: The ID of the vector database you want to retrieve.

Returns:

vector databaseVectorDatabase: The requested vector database.

Return type:

classmethod list(use_case=None, playground=None, search=None, sort=None, completed_only=None)

List all vector databases associated with a specific use case available to the user.

Parameters:

use_caseOptional[UseCaseLike], optional: The returned vector databases are filtered to those associated with a specific Use Case or Cases if specified or can be inferred from the Context. Accepts either the entity or the ID.
playgroundOptional[Union[Playground, str]], optional: The returned vector databases are filtered to those associated with a specific playground if it is specified. Accepts either the entity or the ID.
searchstr, optional: String for filtering vector databases. Vector databases that contain the string in name will be returned. If not specified, all vector databases will be returned.
sortstr, optional: Property to sort vector databases by. Prefix the attribute name with a dash to sort in descending order, e.g. sort=’-creationDate’. Currently supported options are listed in ListVectorDatabasesSortQueryParams but the values can differ with different platform versions. By default, the sort parameter is None which will result in vector databases being returned in order of creation time descending.
completed_onlybool, optional: A filter to retrieve only vector databases that have been successfully created. By default, all vector databases regardless of execution status are retrieved.

Returns:

vectorbaseslist[VectorDatabase]: A list of vector databases available to the user.

Return type:

List[VectorDatabase]

update(name)

Update the vector database.

Parameters:

namestr: The new name for the vector database.

Returns:

vector databaseVectorDatabase: The updated vector database.

Return type:

EvaluationDatasetConfiguration

delete()

Delete the vector database.

Return type:: None

classmethod get_supported_text_chunkings()

Get all supported text chunking configurations which includes a set of recommended chunking parameters for each supported embedding model.

Returns:

supported_text_chunkingsSupportedTextChunkings: The supported text chunking configurations.

Return type:

SupportedTextChunkings

download_text_and_embeddings_asset(file_path=None)

Download a parquet file with text chunks and corresponding embeddings created by a vector database.

Parameters:

file_pathstr, optional: File path to save the asset. By default, it saves in the current directory autogenerated by server name.

Return type:

None

class datarobot.models.genai.vector_database.SupportedRetrievalSetting(name, type, description, title, default, minimum, maximum, enum, settings, group_id)

A single supported retrieval setting.

Attributes:

name: str: The name of the setting.
type: str or list[str]: The type of the setting.
description: str: The description of the setting.
title: str: The title of the setting.
default: str, int, bool, or None: The default value of the setting.
minimum: int or None: The minimum value of the setting.
maximum: int or None: The maximum value of the setting.
enum: list[str] or None: The enum values of the setting.
settings: list[SupportedRetrievalSetting] or None: The supported retriever settings.
group_id: str or None: The group ID of the setting.

class datarobot.models.genai.vector_database.SupportedRetrievalSettings(settings)

Supported retrieval settings.

Attributes:

settingslist[dict]: The supported retriever settings.

class datarobot.models.genai.vector_database.VectorDatabaseDatasetExportJob(job_id, vector_database_id, export_dataset_id)

Response for the vector database dataset export job.

Attributes:

job_idstr: ID of the export job.
vector_database_idstr: ID of the vector database.
export_dataset_idstr: ID of the exported dataset.

class datarobot.models.genai.chat_prompt.Citation(text, source=None, similarity_score=None, metadata=None)

Citation for documents retrieved from a vector database.

Attributes:

textstr: The text retrieved from a vector database.
sourcestr or None, optional: The source of the retrieved text.
similarity_score:: The similarity score between the citation and the user prompt.
metadata: dict or None, optional: Additional metadata for the citation.

class datarobot.models.genai.chat_prompt.ResultMetadata(output_token_count, input_token_count, total_token_count, estimated_docs_token_count, latency_milliseconds, feedback_result, metrics, final_prompt=None, error_message=None, cost=None)

Metadata for the result of a chat prompt submission.

Attributes:

output_token_countint: The number of tokens in the output.
input_token_countint: The number of tokens in the input. This includes the chat history and documents retrieved from a vector database, if any.
total_token_countint: The total number of tokens processed.
estimated_docs_token_countint: The estimated number of tokens from the documents retrieved from a vector database, if any.
latency_millisecondsint: The latency of the chat prompt submission in milliseconds.
feedback_resultFeedbackResult: The lists of user_ids providing positive and negative feedback.
metricsMetricMetadata: The evaluation metrics for this prompt.
final_promptOptional[Union[str, dict]], optional: Representation of the final prompt sent to the LLM.
error_messagestr or None, optional: The error message from the LLM response.
costfloat or None, optional: The cost of the chat prompt submission.

class datarobot.models.genai.llm_blueprint.VectorDatabaseSettings(max_documents_retrieved_per_prompt=None, max_tokens=None, retriever=VectorDatabaseRetrievers.SINGLE_LOOKUP_RETRIEVER, add_neighbor_chunks=False)

Settings for a DataRobot GenAI vector database associated with an LLM blueprint.

Attributes:

max_documents_retrieved_per_promptint or None, optional: The maximum number of documents to retrieve for each prompt.
max_tokensint or None, optional: The maximum number of tokens to retrieve for each document.
retriever: VectorDatabaseRetrievers: The vector database retriever name.
add_neighbor_chunks: Whether to add neighboring documents to the retrieved documents.

class datarobot.models.genai.vector_database.ChunkingParameters(embedding_model, chunking_method, chunk_size, chunk_overlap_percentage, separators, custom_chunking=False, embedding_validation=None)

Parameters defining how documents are split and embedded.

Attributes:

embedding_modelOptional[str]: Name of the text embedding model. Currently supported options are listed in VectorDatabaseEmbeddingModel but the values can differ with different platform versions.
chunking_methodstr: Name of the method to split dataset documents. Currently supported options are listed in VectorDatabaseChunkingMethod but the values can differ with different platform versions.
chunk_sizeint: Size of each text chunk in number of tokens.
chunk_overlap_percentageint: Overlap percentage between chunks.
separatorslist[str]: Strings used to split documents into text chunks.
embedding_validationOptional[CustomModelEmbeddingValidation, SupportedCustomModelEmbedding, str]: ID or object for custom embedding validation.
custom_chunkingbool: Determines if the chunking is custom. With custom chunking, dataset rows are not split into chunks automatically; instead, the user provides the chunks.

class datarobot.models.genai.prompt_trace.PromptTrace(timestamp, user, use_case_id, llm_blueprint_id, llm_blueprint_name, text, execution_status, llm_name=None, llm_vendor=None, llm_license=None, chat_prompt_id=None, comparison_prompt_id=None, llm_settings=None, chat_name=None, chat_id=None, vector_database_id=None, vector_database_settings=None, result_metadata=None, result_text=None, confidence_scores=None, evaluation_dataset_configuration_id=None)

Prompt trace contains aggregated information about a prompt execution.

Attributes:

timestampstr: The timestamp of the trace (ISO 8601 formatted).
userdict: The user who submitted the prompt.
chat_prompt_idstr: The ID of the chat prompt associated with the trace.
use_case_idstr: The ID of the Use Case the playground is in.
comparison_prompt_idstr: The ID of the comparison prompts associated with the trace.
llm_blueprint_idstr: The ID of the LLM blueprint that the prompt was submitted to.
llm_blueprint_name: str: The name of the LLM blueprint.
llm_namestr: The name of the LLM in the LLM blueprint.
llm_vendorstr: The vendor name of the LLM.
llm_licensestr: What type of license the LLM has.
llm_settingsdict or None: The LLM settings for the LLM blueprint. The specific keys allowed and the constraints on the values are defined in the response from LLMDefinition.list, but this typically has dict fields. Either: - system_prompt - The system prompt that influences the LLM responses. - max_completion_length - The maximum number of tokens in the completion. - temperature - Controls the variability in the LLM response. - top_p - Sets whether the model considers next tokens with top_p probability mass. or - system_prompt - The system prompt that influences the LLM responses. - validation_id - The ID of the external model LLM validation. - external_llm_context_size - The external LLM’s context size, in tokens, for external model-based LLM blueprints.
chat_name: str or None: The name of the chat associated with the Trace.
chat_id: str or None: The ID of the chat associated with the Trace.
vector_database_idstr or None: ID of the vector database associated with the LLM blueprint, if any.
vector_database_settingsVectorDatabaseSettings or None: The settings for the vector database associated with the LLM blueprint, if any.
result_metadataResultMetadata or None: Metadata for the result of the prompt submission.
result_text: str or None: The result text from the prompt submission.
confidence_scores: ConfidenceScores or None: The confidence scores if there is a vector database associated with the prompt.
text: str: The prompt text submitted to the LLM.
execution_status: str: The execution status of the chat prompt.
evaluation_dataset_configuration_id: str or None: The ID of the evaluation dataset configuration associated with the trace.

classmethod list(playground)

List all prompt traces for a playground.

Parameters:

playground: str: The ID of the playground to list prompt traces for.

Returns:

prompt_traces: list[PromptTrace]: List of prompt traces for the playground.

Return type:

List[PromptTrace]

classmethod export_to_ai_catalog(playground)

Export prompt traces to AI Catalog as a CSV.

Parameters:

playground: str: The ID of the playground to export prompt traces for.

Returns:

status_url: str: The URL where the status of the job can be monitored

Return type:

Any

class datarobot.models.genai.insights_configuration.InsightsConfiguration(insight_name, insight_type=None, deployment_id=None, model_id=None, sidecar_model_metric_validation_id=None, custom_metric_id=None, evaluation_dataset_configuration_id=None, cost_configuration_id=None, result_unit=None, ootb_metric_id=None, ootb_metric_name=None, guard_conditions=None, moderation_configuration=None, execution_status=None, error_message=None, error_resolution=None, nemo_metric_id=None, llm_id=None, custom_model_llm_validation_id=None, stage=None, aggregation_types=None, sidecar_model_metric_metadata=None, guard_template_id=None, guard_configuration_id=None, model_package_registered_model_id=None)

Configuration information for a specific insight.

Attributes:

insight_namestr: The name of the insight.
insight_typeInsightTypes, optional: The type of the insight.
deployment_idstr, optional: The deployment ID the insight is applied to.
model_idstr, optional: The model ID for the insight.
sidecar_model_metric_validation_idstr, optional: Validation ID for the sidecar model metric.
custom_metric_idstr, optional: The ID for a custom model metric.
evaluation_dataset_configuration_idstr, optional: The ID for the evaluation dataset configuration.
cost_configuration_idstr, optional: The ID for the cost configuration information.
result_unitstr, optional: The unit of the result, for example “USD”.
ootb_metric_idstr, optional: The ID of the Datarobot-provided metric that does not require additional configuration.
ootb_metric_namestr, optional: The name of the Datarobot-provided metric that does not require additional configuration.
guard_conditionslist[dict], optional: The guard conditions to be used with the insight.
moderation_configurationdict, optional: The moderation configuration for the insight.
execution_statusstr, optional: The execution status of the insight.
error_messagestr, optional: The error message for the insight, for example if it is missing specific configuration for deployed models.
error_resolutionstr, optional: An indicator of which field must be edited to resolve an error state.
nemo_metric_idstr, optional: The ID for the NEMO metric.
llm_idstr, optional: The LLM ID for OOTB metrics that use LLMs.
custom_model_llm_validation_idstr, optional: The ID for the custom model LLM validation if using a custom model LLM for OOTB metrics.
aggregation_typeslist[str], optional: The aggregation types to be used for the insight.
stagestr, optional: The stage (prompt or response) when the metric is calculated.
sidecar_model_metric_metadatadict, optional: Metadata specific to sidecar model metrics.
guard_template_idstr, optional: The ID for the guard template that applies to the insight.
guard_configuration_idstr, optional: The ID for the guard configuration that applies to the insight.

class datarobot.models.genai.insights_configuration.SupportedInsights(insight_name, insight_type=None, deployment_id=None, model_id=None, sidecar_model_metric_validation_id=None, custom_metric_id=None, evaluation_dataset_configuration_id=None, cost_configuration_id=None, result_unit=None, ootb_metric_id=None, ootb_metric_name=None, guard_conditions=None, moderation_configuration=None, execution_status=None, error_message=None, error_resolution=None, nemo_metric_id=None, llm_id=None, custom_model_llm_validation_id=None, stage=None, aggregation_types=None, sidecar_model_metric_metadata=None, guard_template_id=None, guard_configuration_id=None, model_package_registered_model_id=None)

Supported insights configurations for a given use case.

classmethod list(use_case_id)

Get a list of all supported insights that can be used within a given Use Case.

Parameters:

use_case_id: str: The ID of the Use Case to list supported insights for.

Returns:

insights: list[InsightsConfiguration]: A list of supported insights.

Return type:

List[InsightsConfiguration]

class datarobot.models.genai.insights_configuration.Insights(playground_id, insights_configuration, creation_date, creation_user_id, last_update_date, last_update_user_id, tenant_id)

The insights configured for a playground.

Attributes:

playground_idstr: The ID of the playground the insights are configured for.
insights_configurationlist[InsightsConfiguration]: The insights configuration for the playground.
creation_datestr: The date the insights were configured.
creation_user_idstr: The ID of the user who created the insights.
last_update_datestr: The date the insights were last updated.
last_update_user_idstr: The ID of the user who last updated the insights.
tenant_idstr: The tenant ID that applies to the record.

classmethod get(playground, with_aggregation_types_only=False)

Get the insights configuration for a given playground.

Parameters:

playground: str|Playground: The ID of the playground to get insights for.
with_aggregation_types_only: bool, optional: If True, only return the aggregation types for the insights.

Returns:

insights: Insights: The insights configuration for the playground.

Return type:

Insights

classmethod create(playground, insights_configuration, use_case)

Create a new insights configuration for a given playground.

Parameters:

playground: str: The ID of the playground to create insights for.
insights_configuration: list[InsightsConfiguration]: The insights configuration for the playground.
use_case_id: str: The Use Case ID to the playground is a part of.

Returns:

insights: Insights: The created insights configuration.

Return type:

Insights

class datarobot.models.genai.cost_metric_configurations.LLMCostConfiguration(input_token_price, reference_input_token_count, output_token_price, reference_output_token_count, currency_code, llm_id, custom_model_llm_validation_id=None)

Cost configuration for a specific LLM model; used for cost metric calculation. Price-per-token is price/reference token count.

Attributes:

input_token_price (float): The price of the input token.
reference_input_token_count (int): The reference input token count.
output_token_price (float): The price of the output token.
reference_output_token_count (int): The reference output token count.
currency_code (str): The currency code.
llm_id (str): The LLM ID.
custom_model_llm_validation_id (Optional[str]): The custom model LLM validation ID if llm_id is custom-model.

class datarobot.models.genai.cost_metric_configurations.CostMetricConfiguration(cost_configuration_id, playground_id, use_case_id, cost_metric_configurations, name)

Cost metric configuration for a use case.

Attributes:

cost_configuration_id (str): The cost configuration ID.
use_case_id (str): The use case ID.
cost_metric_configurations (List[LLMCostConfiguration]): The list of LLM cost configurations.

classmethod get(cost_metric_configuration_id)

Get cost metric configuration by ID.

Return type:: CostMetricConfiguration

update(cost_metric_configurations, name=None)

Update the cost configurations.

Return type:: CostMetricConfiguration

classmethod create(use_case_id, playground_id, name, cost_metric_configurations)

Create a new cost metric configuration.

Return type:: CostMetricConfiguration

delete()

Delete the cost metric configuration.

Return type:: None

class datarobot.models.genai.evaluation_dataset_configuration.EvaluationDatasetConfiguration(id, name, size, rows_count, use_case_id, dataset_id, dataset_name, prompt_column_name, user_name, creation_user_id, creation_date, tenant_id, execution_status, playground_id=None, response_column_name=None, correctness_enabled=None, error_message=None)

An evaluation dataset configuration used to evaluate the performance of LLMs.

Attributes:

idstr: The evaluation dataset configuration ID.
namestr: The name of the evaluation dataset configuration.
sizeint: The size of the evaluation dataset (in bytes).
rows_countint: The row count of the evaluation dataset.
use_case_idstr: The ID of the Use Case associated with the evaluation dataset configuration.
playground_idOptional[str]: The ID of the playground associated with the evaluation dataset configuration.
dataset_idstr: The ID of the evaluation dataset.
dataset_namestr: The name of the evaluation dataset.
prompt_column_namestr: The name of the dataset column containing the prompt text.
response_column_nameOptional[str]: The name of the dataset column containing the response text.
user_namestr: The name of the user who created the evaluation dataset configuration.
correctness_enabledOptional[bool]: Whether correctness is enabled for the evaluation dataset configuration.
creation_user_idstr: The ID of the user who created the evaluation dataset configuration.
creation_datestr: The creation date of the evaluation dataset configuration (ISO-8601 formatted).
tenant_idstr: The ID of the DataRobot tenant this evaluation dataset configuration belongs to.
execution_statusstr: The execution status of the evaluation dataset configuration.
error_messageOptional[str]: The error message associated with the evaluation dataset configuration.

classmethod get(id)

Get an evaluation dataset configuration by ID.

Parameters:

id: str: The evaluation dataset configuration ID to fetch.

Returns:

evaluation_dataset_configuration: EvaluationDatasetConfiguration: The evaluation dataset configuration.

Return type:

classmethod list(use_case_id, playground_id=None, evaluation_dataset_configuration_id=None, offset=0, limit=100, sort=None, search=None, correctness_only=False, completed_only=False)

List all evaluation dataset configurations for a Use Case.

Parameters:

use_case_id: str: The ID of the Use Case that evaluation datasets are returned for.
playground_id: str, optional: The ID of the playground that evaluation datasets are returned for. Default is None.
evaluation_dataset_configuration_id: str, optional: The ID of the evaluation dataset configuration to fetch. Default is None.
offset: int, optional: The offset to start fetching evaluation datasets from. Default is 0.
limit: int, optional: The maximum number of evaluation datasets to return. Default is 100.
sort: str, optional: The order of return for evaluation datasets. Default is None, which returns sorting by creation time.
search: str, optional: A search term that filters results so that only evaluation datasets with names matching the string are returned. Default is None.
correctness_only: bool, optional: Whether to return only completed datasets (particularly applicable to completion of generated synthetic datasets). Default is False.
completed_only: bool, optional: Whether to return only completed datasets. Default is False.

Returns:

evaluation_dataset_configurations: List[EvaluationDatasetConfiguration]: A list of evaluation dataset configurations.

Return type:

List[EvaluationDatasetConfiguration]

classmethod create(name, use_case_id, dataset_id, prompt_column_name, playground_id, is_synthetic_dataset=False, response_column_name=None)

Create an evaluation dataset configuration for an existing dataset.

Parameters:

name: str: The name of the evaluation dataset configuration.
use_case_id: str: The Use Case ID that the evaluation dataset configuration will be added to.
dataset_id: str: An ID, to add to the configuration, that identifies the evaluation dataset.
playground_id: str: The ID of the playground that the evaluation dataset configuration will be added to. Default is None.
prompt_column_name: str: The name of the prompt column in the dataset.
response_column_name: str: The name of the response column in the dataset.
is_synthetic_dataset: bool: Whether the evaluation dataset is synthetic.

Returns:

evaluation_dataset_configurationEvaluationDatasetConfiguration: The created evaluation dataset configuration.

Return type:

EvaluationDatasetConfiguration

update(name=None, dataset_id=None, prompt_column_name=None, response_column_name=None)

Update the evaluation dataset configuration.

Parameters:

name: str, optional: The name of the evaluation dataset configuration.
dataset_id: str, optional: The ID of the dataset used in this configuration.
prompt_column_namestr, optional: The name of the prompt column in the dataset.
response_column_namestr, optional: The name of the response column in the dataset.

Returns:

evaluation_dataset_configurationEvaluationDatasetConfiguration: The updated evaluation dataset configuration.

Return type:

EvaluationDatasetConfiguration

delete()

Delete the evaluation dataset configuration.

Returns:

None

Return type:

None

class datarobot.models.genai.evaluation_dataset_metric_aggregation.EvaluationDatasetMetricAggregation(llm_blueprint_id, evaluation_dataset_configuration_id, ootb_dataset_name, metric_name, deployment_id, dataset_id, dataset_name, chat_id, chat_name, aggregation_value, aggregation_type, creation_date, creation_user_id, tenant_id)

Information about an evaluation dataset metric aggregation job.: This job runs a metric against LLMs using an evaluation dataset and aggregates the results.

Attributes:

llm_blueprint_idstr: The LLM blueprint ID.
evaluation_dataset_configuration_idstr: The evaluation dataset configuration ID.
ootb_dataset_namestr | None: The name of the Datarobot-provided dataset that does not require additional configuration..
metric_namestr: The name of the metric.
deployment_idstr | None: A deployment ID if the evaluation was run against a deployment.
dataset_idstr | None: The ID of the dataset used in the evaluation.
dataset_namestr | None: The name of the dataset used in the evaluation.
chat_idstr: The ID of the chat created to run the evaluation.
chat_namestr: The name of the chat that was created to run the evaluation.
aggregation_valuefloat | List[Dict[str, float]]: The aggregated metric result.
aggregation_typeAggregationType: The type of aggregation used for the metric results.
creation_datestr: The date the evaluation job was created.
creation_user_idstr: The ID of the user who created the evaluation job.
tenant_idstr: The ID of the tenant that owns the evaluation job.

classmethod create(chat_name, llm_blueprint_ids, evaluation_dataset_configuration_id, insights_configuration)

Create a new evaluation dataset metric aggregation job. The job will run the specified metric for the specified LLM blueprint IDs using the prompt-response pairs in the evaluation dataset.

Parameters:

chat_namestr: The name of the chat that will be created to run the evaluation in.
llm_blueprint_idsList[str]: The LLM blueprint IDs to evaluate.
evaluation_dataset_configuration_idstr: The ID evaluation dataset configuration to use during the evaluation.
insights_configurationList[InsightsConfiguration]: The insights configurations to use during the evaluation.

Returns:

str: The ID of the evaluation dataset metric aggregation job.

Return type:

Any

classmethod list(llm_blueprint_ids=None, chat_ids=None, evaluation_dataset_configuration_ids=None, metric_names=None, aggregation_types=None, current_configuration_only=False, sort=None, offset=0, limit=100, non_errored_only=True)

List evaluation dataset metric aggregations. The results will be filtered by the provided LLM blueprint IDs and chat IDs.

Parameters:

llm_blueprint_idsList[str]: The LLM blueprint IDs to filter on.
chat_idsList[str]: The chat IDs to filter on.
evaluation_dataset_configuration_idsList[str]: The evaluation dataset configuration IDs to filter on.
metric_namesList[str]: The metric names to filter on.
aggregation_typesList[str]: The aggregation types to filter on.
current_configuration_onlybool, optional: If True, only results that are associated with the current configuration of the LLM blueprint will be returned. Defaults to False.
sortstr, optional: The field to sort on. Defaults to None.
offsetint, optional: The offset to start at. Defaults to 0.
limitint, optional: The maximum number of results to return. Defaults to 100.
non_errored_onlybool, optional: If True, only results that did not encounter an error will be returned. Defaults to True.

Returns:

List[EvaluationDatasetMetricAggregation]: A list of evaluation dataset metric aggregations.

Return type:

List[EvaluationDatasetMetricAggregation]

classmethod delete(llm_blueprint_ids, chat_ids)

Delete the associated evaluation dataset metric aggregations. Either llm_blueprint_ids or chat_ids must be provided. If both are provided, only results matching both will be removed.

Parameters:

llm_blueprint_idsList[str]: The LLM blueprint IDs to filter on.
chat_idsList[str]: The chat IDs to filter on.

Return type:

None

class datarobot.models.genai.synthetic_evaluation_dataset_generation.SyntheticEvaluationDataset(dataset_id, prompt_column_name, response_column_name)

A synthetically generated evaluation dataset for LLMs.

Attributes:

dataset_id (str): The ID of the dataset.
prompt_column_name (str): The name of the prompt column in the dataset.
response_column_name (str): The name of the response column in the dataset.

classmethod create(llm_id, vector_database_id, llm_settings=None, dataset_name=None, language=None)

Create a synthetic evaluation dataset generation job. This will create a synthetic dataset to be used for evaluation of a language model.

Parameters:

llm_id (str): The ID of the language model that will be used to generate the dataset.
llm_settings (Dict[str, Optional[Union[bool, int, float, str]]]): The settings to use for the language: model used for dataset generation.
vector_database_id (str): The ID of the vector database to use for dataset generation.
dataset_name (str): The name to use for the dataset, otherwise this will be autogenerated.
language (str): The language to use for dataset generation. English will be used if this is not specified.

Returns:

SyntheticEvaluationDataset: Reference to the synthetic evaluation dataset that was created.

Return type:

SyntheticEvaluationDataset

class datarobot.models.genai.sidecar_model_metric.SidecarModelMetricValidation(id, playground_id, prompt_column_name, deployment_id, model_id, validation_status, deployment_access_data, tenant_id, name, creation_date, user_id, deployment_name, user_name, use_case_id, prediction_timeout, error_message, citations_prefix_column_name, response_column_name, target_column_name, expected_response_column_name)

A sidecar model metric validation for LLMs.

Attributes:

idstr: The ID of the sidecar model metric validation.
prompt_column_namestr: The name of the prompt column for the sidecar model.
deployment_idstr: The ID of the deployment associated with the sidecar model.
model_idstr: The ID of the sidecar model.
validation_statusstr: The status of the validation job.
deployment_access_datadict: Data that will be used for accessing deployment prediction server. Only available for deployments that passed validation. Dict fields: - prediction_api_url - URL for deployment prediction server. - datarobot_key - first of 2 auth headers for the prediction server. - authorization_header - second of 2 auth headers for the prediction server. - input_type - Either JSON or CSV - the input type that the model expects. - model_type - Target type of the deployed custom model.
tenant_idstr: The ID of the tenant that created the sidecar model metric validation.
namestr: The name of the sidecar model metric.
creation_datestr: The date the sidecar model metric validation was created.
user_idstr: The ID of the user that created the sidecar model metric validation.
deployment_namestr: The name of the deployment associated with the sidecar model.
user_namestr: The name of the user that created the sidecar model metric validation.
use_case_idstr: The ID of the use case associated with the sidecar model metric validation.
prediction_timeoutint: The timeout in seconds for the prediction API used in this sidecar model metric validation.
error_messagestr: Additional information for errored validation.
citations_prefix_column_namestr: The name of the prefix in the citations column for the sidecar model.
response_column_namestr: The name of the response column for the sidecar model.
expected_response_column_namestr: The name of the expected response column for the sidecar model.
target_column_namestr: The name of the target column for the sidecar model.

classmethod create(deployment_id, name, prediction_timeout, model_id=None, use_case_id=None, playground_id=None, prompt_column_name=None, target_column_name=None, response_column_name=None, citation_prefix_column_name=None, expected_response_column_name=None)

Create a sidecar model metric validation.

Parameters:

deployment_idstr: The ID of the deployment to validate.
namestr: The name of the validation.
prediction_timeoutint: The timeout in seconds for the prediction API used in this validation.
model_idstr, optional: The ID of the model to validate.
use_case_idstr, optional: The ID of the use case associated with the validation.
playground_idstr, optional: The ID of the playground associated with the validation.
prompt_column_namestr, optional: The name of the prompt column for the sidecar model.
target_column_namestr, optional: The name of the target column for the sidecar model.
response_column_namestr, optional: The name of the response column for the sidecar model.
citation_prefix_column_namestr, optional: The name of the prefix for citations column for the sidecar model.
expected_response_column_namestr, optional: The name of the expected response column for the sidecar model.

Returns:

SidecarModelMetricValidation: The created sidecar model metric validation.

Return type:

classmethod list(use_case_ids=None, offset=None, limit=None, search=None, sort=None, completed_only=True, deployment_id=None, model_id=None, prompt_column_name=None, target_column_name=None, citation_prefix_column_name=None)

List sidecar model metric validations.

Parameters:

use_case_idsList[str], optional: The IDs of the use cases to filter by.
offsetint, optional: The number of records to skip.
limitint, optional: The maximum number of records to return.
searchstr, optional: The search string.
sortstr, optional: The sort order.
completed_onlybool, optional: Whether to return only completed validations.
deployment_idstr, optional: The ID of the deployment to filter by.
model_idstr, optional: The ID of the model to filter by.
prompt_column_namestr, optional: The name of the prompt column to filter by.
target_column_namestr, optional: The name of the target column to filter by.
citation_prefix_column_namestr, optional: The name of the prefix for citations column to filter by.

Returns:

List[SidecarModelMetricValidation]: The list of sidecar model metric validations.

Return type:

List[SidecarModelMetricValidation]

classmethod get(validation_id)

Get a sidecar model metric validation by ID.

Parameters:

validation_idstr: The ID of the validation to get.

Returns:

SidecarModelMetricValidation: The sidecar model metric validation.

Return type:

revalidate()

Revalidate the sidecar model metric validation.

Returns:

SidecarModelMetricValidation: The sidecar model metric validation.

Return type:

update(name=None, prompt_column_name=None, target_column_name=None, response_column_name=None, expected_response_column_name=None, citation_prefix_column_name=None, deployment_id=None, model_id=None, prediction_timeout=None)

Update the sidecar model metric validation.

Parameters:

namestr, optional: The name of the validation.
prompt_column_namestr, optional: The name of the prompt column for the sidecar model.
target_column_namestr, optional: The name of the target column for the sidecar model.
response_column_namestr, optional: The name of the response column for the sidecar model.
expected_response_column_namestr, optional: The name of the expected response column for the sidecar model.
citation_prefix_column_namestr, optional: The name of the prefix for citations column for the sidecar model.
deployment_idstr, optional: The ID of the deployment to validate.
model_idstr, optional: The ID of the model to validate.
prediction_timeoutint, optional: The timeout in seconds for the prediction API used in this validation.

Returns:

SidecarModelMetricValidation: The updated sidecar model metric validation.

Return type: