Datasets

class datarobot.models.Dataset

Represents a Dataset returned from the api/v2/datasets/ endpoints.

Variables:

id (string) – The ID of this dataset.
name (string) – The name of this dataset in the catalog.
is_latest_version (bool) – Whether this dataset version is the latest version of this dataset.
version_id (string) – The object ID of the catalog_version the dataset belongs to.
categories (list(string)) – An array of strings describing the intended use of the dataset. The supported options are “TRAINING” and “PREDICTION”.
created_at (string) – The date when the dataset was created.
created_by (string, optional) – Username of the user who created the dataset.
is_snapshot (bool) – Whether the dataset version is an immutable snapshot of data that was previously retrieved and saved to DataRobot.
data_persisted (Optional[bool]) – If true, user is allowed to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.) and download data. If false, download is not allowed and only the data schema (feature names and types) will be available.
is_data_engine_eligible (bool) – Whether this dataset can be a data source of a data engine query.
processing_state (string) – Current ingestion process state of the dataset.
row_count (Optional[int]) – The number of rows in the dataset.
size (Optional[int]) – The size of the dataset as a CSV in bytes.
sample_size (dict, optional) – The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value is {‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.

get_uri()

Returns:: url – Permanent static hyperlink to this dataset in AI Catalog.
Return type:: str

classmethod upload(source)

This method covers Dataset creation from local materials (file and DataFrame) and a URL.

Parameters:: source (str, pd.DataFrame or file object) – Pass a URL, filepath, file or DataFrame to create and return a Dataset.
Returns:: response – The Dataset created from the uploaded data source.
Return type:: Dataset
Raises:: InvalidUsageError – If the source parameter can’t be determined to be a URL, filepath, file, or DataFrame.

Examples

# Upload a local file
dataset_one = Dataset.upload("./data/examples.csv")

# Create a dataset via URL
dataset_two = Dataset.upload(
    "https://raw.githubusercontent.com/curran/data/gh-pages/dbpedia/cities/data.csv"
)

# Create dataset with a pandas Dataframe
dataset_three = Dataset.upload(my_df)

# Create dataset using a local file
with open("./data/examples.csv", "rb") as file_pointer:
    dataset_four = Dataset.create_from_file(filelike=file_pointer)

classmethod create_from_file(cls, file_path=None, filelike=None, categories=None, read_timeout=600, max_wait=600, *, use_cases=None)

A blocking call that creates a new Dataset from a file. Returns the dataset once it has been successfully uploaded and processed.

Warning: This function doesn’t clean up its open files. If you pass a filelike, you are responsible for closing it. If you pass a file_path, this will create a file object from the file_path but won’t close it.

Parameters:

file_path (string, optional) – The path to the file. This will create a file object pointing to that file but won’t close it.
filelike (file, optional) – An open and readable file object.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout (Optional[int]) – The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete.
max_wait (Optional[int]) – Time in seconds after which dataset creation is considered unsuccessful.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new Dataset to. Must be a kwarg.

Returns:

response – A fully armed and operational Dataset.

Return type:

Dataset

classmethod create_from_in_memory_data(cls, data_frame=None, records=None, categories=None, read_timeout=600, max_wait=600, fname=None, *, use_cases=None)

A blocking call that creates a new Dataset from in-memory data. Returns the dataset once it has been successfully uploaded and processed.

The data can be either a pandas DataFrame or a list of dictionaries with identical keys.

Parameters:

data_frame (DataFrame, optional) – The data frame to upload.
records (list[dict], optional) – A list of dictionaries with identical keys to upload.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout (Optional[int]) – The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete.
max_wait (Optional[int]) – Time in seconds after which dataset creation is considered unsuccessful.
fname (string, optional) – The filename. Defaults to data.csv.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns:

response – The Dataset created from the uploaded data.

Return type:

Dataset

Raises:

InvalidUsageError – If you don’t pass a DataFrame or a list of records.

classmethod create_from_url(cls, url, do_snapshot=None, persist_data_after_ingestion=None, categories=None, sample_size=None, max_wait=600, *, use_cases=None)

A blocking call that creates a new Dataset from data stored at a URL. Returns the dataset once it has been successfully uploaded and processed.

Parameters:

url (string) – The URL to use as the source of data for the dataset being created.
do_snapshot (Optional[bool]) – If unset, uses the server default: True. If true, creates a snapshot dataset; if false, creates a remote dataset. Creating snapshots from non-file sources may be disabled by the permission, Disable AI Catalog Snapshots.
persist_data_after_ingestion (Optional[bool]) – If unset, uses the server default: True. If true, will enforce saving all data (for download and sampling) and will allow a user to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.). If false, won’t enforce saving data. The data schema (feature names and types) will still be available. Specifying this parameter to false and doSnapshot to true will result in an error.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
sample_size (dict, optional) – The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value would be: {‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.
max_wait (Optional[int]) – Time in seconds after which dataset creation is considered unsuccessful.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns:

response – The Dataset created from the uploaded data.

Return type:

Dataset

classmethod create_from_project(cls, project_id, categories=None, max_wait=600, *, use_cases=None)

A blocking call that creates a new dataset from project data. Returns the dataset once it has been successfully created.

Parameters:

project_id (string) – The project to create the dataset from.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait (Optional[int]) – Time in seconds after which dataset creation is considered unsuccessful.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns:

response – The dataset created from the project dataset.

Return type:

Dataset

classmethod create_from_datastage(cls, datastage_id, categories=None, max_wait=600, *, use_cases=None)

A blocking call that creates a new Dataset from data stored as a DataStage. Returns the dataset once it has been successfully uploaded and processed.

Parameters:

datastage_id (string) – The ID of the DataStage to use as the source of data for the dataset being created.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait (Optional[int]) – Time in seconds after which dataset creation is considered unsuccessful.

Returns:

response – The Dataset created from the uploaded data.

Return type:

Dataset

classmethod create_from_data_source(cls, data_source_id, username=None, password=None, do_snapshot=None, persist_data_after_ingestion=None, categories=None, credential_id=None, use_kerberos=None, credential_data=None, sample_size=None, max_wait=600, *, use_cases=None)

A blocking call that creates a new Dataset from data stored at a DataSource. Returns the dataset once it has been successfully uploaded and processed.

Added in version v2.22.

Parameters:

data_source_id (string) – The ID of the DataSource to use as the source of data.
username (string, optional) – The username for database authentication.
password (string, optional) – The password (in cleartext) for database authentication. The server encrypts the password in the scope of the HTTP request and never saves or stores it.
do_snapshot (Optional[bool]) – If unset, uses the server default: True. If true, creates a snapshot dataset; if false, creates a remote dataset. Creating snapshots from non-file sources requires may be disabled by the permission, Disable AI Catalog Snapshots.
persist_data_after_ingestion (Optional[bool]) – If unset, uses the server default: True. If true, will enforce saving all data (for download and sampling) and will allow a user to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.). If false, won’t enforce saving data. The data schema (feature names and types) will still be available. Specifying this parameter to false and doSnapshot to true will result in an error.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
credential_id (string, optional) – The ID of the set of credentials to use instead of user and password. Note that with this change, username and password will become optional.
use_kerberos (Optional[bool]) – If unset, uses the server default: False. If true, use kerberos authentication for database authentication.
credential_data (dict, optional) – The credentials to authenticate with the database, to use instead of user/password or credential ID.
sample_size (dict, optional) – The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value would be: {‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns:

response – The Dataset created from the uploaded data.

Return type:

Dataset

classmethod create_from_query_generator(cls, generator_id, dataset_id=None, dataset_version_id=None, max_wait=600, *, use_cases=None)

A blocking call that creates a new Dataset from the query generator. Returns the dataset once it has been successfully processed. If optional parameters are not specified, the query applies to the dataset_id and dataset_version_id stored in the query generator. If specified, they override the stored dataset_id/dataset_version_id, for example, to prepare a prediction dataset.

Parameters:

generator_id (str) – The ID of the query generator to use.
dataset_id (Optional[str]) – The ID of the dataset to apply the query to.
dataset_version_id (Optional[str]) – The ID of the dataset version to apply the query to. If not specified, the latest version associated with dataset_id (if specified) is used.
max_wait (int) – Optional; the maximum number of seconds to wait before giving up.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new dataset to. Must be a kwarg.

Returns:

response – The Dataset created from the query generator.

Return type:

Dataset

classmethod create_from_recipe(cls, recipe, name=None, do_snapshot=None, persist_data_after_ingestion=None, categories=None, credential=None, credential_id=None, use_kerberos=None, materialization_destination=None, max_wait=600, *, use_cases=None)

A blocking call that creates a new Dataset from the recipe. Returns the dataset once it has been successfully uploaded and processed.

Added in version 3.6.

Returns:: response – The Dataset created from the uploaded data.
Return type:: Dataset

classmethod get(dataset_id)

Get the information about a dataset.

Parameters:: dataset_id (string) – The ID of the dataset.
Returns:: dataset – The queried dataset.
Return type:: Dataset

classmethod delete(dataset_id)

Soft deletes a dataset. You can’t get it or list it or do actions with it, except for un-deleting it.

Parameters:: dataset_id (string) – The ID of the dataset to mark for deletion.
Return type:: None

classmethod un_delete(dataset_id)

Un-deletes a previously deleted dataset. If the dataset wasn’t deleted, nothing happens.

Parameters:: dataset_id (string) – The ID of the dataset to un-delete.
Return type:: None

classmethod list(category=None, filter_failed=None, order_by=None, use_cases=None)

Returns the datasets a user can view.

Parameters:

category (string, optional) – Optional. If specified, only dataset versions that have the specified category will be included in the results. Categories identify the intended use of the dataset; supported categories are “TRAINING” and “PREDICTION”.
filter_failed (Optional[bool]) – If unset, uses the server default: False. Whether datasets that failed during import should be excluded from the results. If True invalid datasets will be excluded.
order_by (string, optional) – If unset, uses the server default: “-created”. Sorting order which will be applied to catalog list, valid options are: - “created” – ascending order by creation datetime; - “-created” – descending order by creation datetime.
use_cases (Union[UseCase, List[UseCase], str, List[str]], optional) – Filter available datasets by a specific Use Case or Cases. Accepts either the entity or the ID. If set to [None], the method filters the project’s datasets by those not linked to a UseCase.

Returns:

A list of datasets the user can view.

Return type:

list[Dataset]

classmethod iterate(offset=None, limit=None, category=None, order_by=None, filter_failed=None, use_cases=None)

Get an iterator for the requested datasets a user can view. This lazily retrieves results. It doesn’t fetch the next page from the server until the iterator exhausts the current page.

Parameters:

offset (Optional[int]) – If set, the iterator skips this many results.
limit (Optional[int]) – Specifies the size of each page retrieved from the server. If unset, uses the server default.
category (string, optional) – Optional. If specified, only dataset versions that have the specified category will be included in the results. Categories identify the intended use of the dataset; supported categories are “TRAINING” and “PREDICTION”.
filter_failed (Optional[bool]) – If unset, uses the server default: False. Whether datasets that failed during import should be excluded from the results. If True invalid datasets will be excluded.
order_by (string, optional) – If unset, uses the server default: “-created”. Sorting order which will be applied to catalog list, valid options are: - “created” – ascending order by creation datetime; - “-created” – descending order by creation datetime.
use_cases (Union[UseCase, List[UseCase], str, List[str]], optional) – Filter available datasets by a specific Use Case or Cases. Accepts either the entity or the ID. If set to [None], the method filters the project’s datasets by those not linked to a UseCase.

Yields:

Dataset – An iterator of the datasets the user can view.

Return type:

Generator[TypeVar(TDataset, bound= Dataset), None, None]

update()

Updates the Dataset attributes in place with the latest information from the server.

Return type:: None

modify(name=None, categories=None)

Modifies the Dataset name and/or categories. Updates the object in place.

Parameters:

name (string, optional) – The new name of the dataset.
categories (list[string], optional) – A list of strings describing the intended use of the dataset. The supported options are “TRAINING” and “PREDICTION.” If any categories were previously specified for the dataset, they will be overwritten. If omitted or None, keep previous categories. To clear them, specify [].

Return type:

None

modify_shared_roles(roles, apply_grant_to_linked_objects=False, operation='updateRoles')

Grant access, remove access, or update roles for users, groups, and organizations who have access to this dataset. A single request can set up to 100 roles.

Parameters:

roles (List[CatalogSharedRoleRequest]) – An array of role requests to modify the roles for.
apply_grant_to_linked_objects (bool) – If true for any users being granted access to the entity, grant the user read access to any linked objects such as DataSources and DataStores that may be used by this entity. Ignored if no such objects are relevant for the entity. This won’t result in access being lowered for a user if the user already has higher access to linked objects than read access. However, if the target user doesn’t have sharing permissions to the linked object, they will be given sharing access without lowering existing permissions. May result in an error if the user making the call doesn’t have sufficient permissions to complete the grant.
operation (str) – The name of the action being taken. The only supported operation is updateRoles.

Return type:

None

Examples

Grant access to a user by name:

>>> from datarobot.enums import TARGET_SHARING_ROLE, SHARING_RECIPIENT_TYPE
>>> from datarobot.models.sharing import CatalogSharedRoleRequest
>>> from datarobot.models.dataset import Dataset

>>> dataset = Dataset.get("my-dataset-id")
>>> dataset.modify_shared_roles(
...     roles=[
...         CatalogSharedRoleRequest(
...             role=TARGET_SHARING_ROLE.EDITOR,
...             share_recipient_type=SHARING_RECIPIENT_TYPE.USER,
...             name="target_user_name",
...         )
...     ],
... )

Remove access to a group by ID:

>>> from datarobot.enums import TARGET_SHARING_ROLE, SHARING_RECIPIENT_TYPE
>>> from datarobot.models.sharing import CatalogSharedRoleRequest
>>> from datarobot.models.dataset import Dataset

>>> dataset = Dataset.get("my-dataset-id")
>>> dataset.modify_shared_roles(
...     roles=[
...         CatalogSharedRoleRequest(
...             role=TARGET_SHARING_ROLE.NO_ROLE,
...             share_recipient_type=SHARING_RECIPIENT_TYPE.GROUP,
...             id="group-id",
...         )
...     ],
... )

share(access_list, apply_grant_to_linked_objects=False)

Changes which users can access this dataset.

Parameters:

access_list (list of SharingAccess) – The modifications to make.
apply_grant_to_linked_objects (bool) – If true for any users being granted access to the dataset, grant the user read access to any linked objects such as DataSources and DataStores that may be used by this dataset. Ignored if no such objects are relevant for dataset, defaults to False.

Return type:

None

Raises:

datarobot.ClientError: – If you don’t have permission to share this dataset, if the user you’re sharing with doesn’t exist, if the same user appears multiple times in the access_list, or if these changes would leave the dataset without an owner.

Examples

Transfer access to the dataset from old_user@datarobot.com to new_user@datarobot.com

from datarobot.enums import SHARING_ROLE
from datarobot.models.dataset import Dataset
from datarobot.models.sharing import SharingAccess

new_access = SharingAccess(
    "new_user@datarobot.com",
    SHARING_ROLE.OWNER,
    can_share=True,
)
access_list = [
    SharingAccess(
        "old_user@datarobot.com",
        SHARING_ROLE.OWNER,
        can_share=True,
        can_use_data=True,
    ),
    new_access,
]

Dataset.get('my-dataset-id').share(access_list)

get_details()

Gets the details for this Dataset

Return type:: DatasetDetails

get_all_features(order_by=None)

Get a list of all the features for this dataset.

Parameters:: order_by (string, optional) – If unset, uses the server default: ‘name’. How the features should be ordered. Can be 'name' or 'featureType'.
Return type:: list[DatasetFeature]

get_shared_roles(id=None, name=None, share_recipient_type=None)

Lists the roles for users, groups, and organizations who have access to this dataset.

Parameters:

id (Optional[str]) – Return only the access control information for an organization, group, or user with this ID.
name (Optional[str]) – Return only the access control information for an organization, group, or user with this name.
share_recipient_type (Optional[SHARING_RECIPIENT_TYPE]) – Only returns results with the given recipient type.

Return type:

List[CatalogSharedRole]

iterate_all_features(offset=None, limit=None, order_by=None)

Get an iterator for the requested features of a dataset. This lazily retrieves results. It doesn’t fetch the next page from the server until the iterator exhausts the current page.

Parameters:

offset (Optional[int]) – If set, the iterator skips this many results.
limit (Optional[int]) – Specifies the size of each page retrieved from the server. If unset, uses the server default.
order_by (string, optional) – If unset, uses the server default: ‘name’. How the features should be ordered. Can be 'name' or 'featureType'.

Yields:

DatasetFeature

Return type:

Generator[DatasetFeature, None, None]

get_featurelists()

Gets the DatasetFeaturelists created on this Dataset

Returns:: feature_lists
Return type:: list[DatasetFeaturelist]

create_featurelist(name, features)

Creates a new dataset featurelist

Parameters:

name (str) – The name of the modeling featurelist to create. Names must be unique within the dataset, or the server will return an error.
features (List[str]) – The names of the features to include in the dataset featurelist. Each feature must be a dataset feature.

Returns:

featurelist – The newly created featurelist.

Return type:

DatasetFeaturelist

Examples

dataset = Dataset.get('1234deadbeeffeeddead4321')
dataset_features = dataset.get_all_features()
selected_features = [feat.name for feat in dataset_features][:5]  # select first five
new_flist = dataset.create_featurelist('Simple Features', selected_features)

get_file(file_path=None, filelike=None)

Retrieves all the originally uploaded data in CSV form. Writes it to either the file or a filelike object that can write bytes.

Provide only one of file_path or filelike, and pass it as a keyword argument (that is, file_path='path-to-write-to'). If you provide a file-like object, you must close it when you’re done.

The user must also have permission to download data.

Parameters:

file_path (string, optional) – The destination to write the file to.
filelike (file, optional) – A file-like object to write to. The object must be able to write bytes. The user is responsible for closing the object.

Return type:

None

get_as_dataframe(low_memory=False)

Retrieves all the originally uploaded data in a pandas DataFrame.

Added in version v3.0.

Parameters:: low_memory (Optional[bool]) – If True, use local files to reduce memory usage which will be slower.
Return type:: pandas.DataFrame

get_projects()

Retrieves projects for this dataset as ProjectLocation named tuples.

Returns:: locations
Return type:: list[ProjectLocation]

get_raw_sample_data()

Retrieves the raw sample data for the dataset as a pandas DataFrame. The raw sample data is a subset of the full dataset.

Added in version v3.10.

Returns:: A DataFrame with the dataset’s raw sample data.
Return type:: pandas.DataFrame

create_project(project_name=None, user=None, password=None, credential_id=None, use_kerberos=None, credential_data=None, *, use_cases=None)

Create a datarobot.models.Project from this dataset

Parameters:

project_name (string, optional) – The name of the project to be created. If not specified, will be “Untitled Project” for database connections, otherwise the project name will be based on the file used.
user (string, optional) – The username for database authentication.
password (string, optional) – The password (in cleartext) for database authentication. The server encrypts the password in the scope of the HTTP request and never saves or stores it.
credential_id (string, optional) – The ID of the set of credentials to use instead of user and password.
use_kerberos (Optional[bool]) – Server default is False. If true, use kerberos authentication for database authentication.
credential_data (dict, optional) – The credentials to authenticate with the database, to use instead of user/password or credential ID.
use_cases (list[UseCase] | UseCase | list[string] | string, optional) – A list of UseCase objects, UseCase object, list of Use Case IDs or a single Use Case ID to add this new Dataset to. Must be a kwarg.

Return type:

Project

classmethod create_version_from_file(dataset_id, file_path=None, filelike=None, categories=None, read_timeout=600, max_wait=600)

A blocking call that creates a new Dataset version from a file. Returns the new dataset version once it has been successfully uploaded and processed.

Warning: This function doesn’t clean up its open files. If you pass a filelike, you are responsible for closing it. If you pass a file_path, this will create a file object from the file_path but won’t close it.

Added in version v2.23.

Parameters:

dataset_id (string) – The ID of the dataset for which a new version is created.
file_path (string, optional) – The path to the file. This will create a file object pointing to that file but won’t close it.
filelike (file, optional) – An open and readable file object.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout (Optional[int]) – The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.

Returns:

response – A fully armed and operational Dataset version.

Return type:

Dataset

classmethod create_version_from_in_memory_data(dataset_id, data_frame=None, records=None, categories=None, read_timeout=600, max_wait=600)

A blocking call that creates a new Dataset version for a dataset from in-memory data. Returns the dataset once it has been successfully uploaded and processed.

The data can be either a pandas DataFrame or a list of dictionaries with identical keys.

Added in version v2.23.

Parameters:

dataset_id (string) – The ID of the dataset for which a new version is created.
data_frame (DataFrame, optional) – The data frame to upload.
records (list[dict], optional) – A list of dictionaries with identical keys to upload.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
read_timeout (Optional[int]) – The maximum number of seconds to wait for the server to respond indicating that the initial upload is complete.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.

Returns:

response – The Dataset version created from the uploaded data.

Return type:

Dataset

Raises:

InvalidUsageError – If you don’t pass a DataFrame or a list of records.

classmethod create_version_from_url(dataset_id, url, categories=None, max_wait=600)

A blocking call that creates a new Dataset from data stored at a URL for a given dataset. Returns the dataset once it has been successfully uploaded and processed.

Added in version v2.23.

Parameters:

dataset_id (string) – The ID of the dataset for which a new version is created.
url (string) – The URL to use as the source of data for the dataset being created.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.

Returns:

response – The Dataset version created from the uploaded data.

Return type:

Dataset

classmethod create_version_from_datastage(dataset_id, datastage_id, categories=None, max_wait=600)

A blocking call that creates a new Dataset from data stored as a DataStage for a given dataset. Returns the dataset once it has been successfully uploaded and processed.

Parameters:

dataset_id (string) – The ID of the dataset for which a new version is created.
datastage_id (string) – The ID of the DataStage to use as the source of data for the dataset being created.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.

Returns:

response – The Dataset version created from the uploaded data.

Return type:

Dataset

classmethod create_version_from_data_source(dataset_id, data_source_id, username=None, password=None, categories=None, credential_id=None, use_kerberos=None, credential_data=None, max_wait=600)

A blocking call that creates a new Dataset from data stored at a DataSource. Returns the dataset once it has been successfully uploaded and processed.

Added in version v2.23.

Parameters:

dataset_id (string) – The ID of the dataset for which a new version is created.
data_source_id (string) – The ID of the DataSource to use as the source of data.
username (string, optional) – The username for database authentication.
password (string, optional) – The password (in cleartext) for database authentication. The server encrypts the password in the scope of the HTTP request and never saves or stores it.
categories (list[string], optional) – An array of strings describing the intended use of the dataset. The current supported options are “TRAINING” and “PREDICTION”.
credential_id (string, optional) – The ID of the set of credentials to use instead of user and password. Note that with this change, username and password will become optional.
use_kerberos (Optional[bool]) – If unset, uses the server default: False. If true, use kerberos authentication for database authentication.
credential_data (dict, optional) – The credentials to authenticate with the database, to use instead of user/password or credential ID.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.

Returns:

response – The Dataset version created from the uploaded data.

Return type:

Dataset

classmethod create_version_from_recipe(dataset_id, recipe, credential=None, credential_id=None, use_kerberos=None, max_wait=600)

A blocking call that creates a new Dataset version from Recipe. Returns the dataset once it has been successfully uploaded and processed.

Added in version v3.8.

Parameters:

dataset_id (string) – The ID of the dataset for which a new version will be created.
recipe (Recipe) – The Recipe to use to create a new dataset version.
credential (Credential, optional) – The credentials to authenticate with the database.
credential_id (string, optional) – The ID of the set of credentials to use instead of Credential object.
use_kerberos (Optional[bool]) – If unset, uses the server default: False. If true, use kerberos authentication for database authentication.
max_wait (Optional[int]) – Time in seconds after which project creation is considered unsuccessful.

Returns:

response – The Dataset version created from the uploaded data.

Return type:

Dataset

classmethod from_data(data)

Instantiate an object of this class using a dict.

Parameters:: data (dict) – Correctly snake_cased keys and their values.
Return type:: TypeVar(T, bound= APIObject)

classmethod from_server_data(data, keep_attrs=None)

Instantiate an object of this class using the data directly from the server, meaning that the keys may have the wrong camel casing

Parameters:

data (dict) – The directly translated dict of JSON from the server. DataRobot has not applied casing fixes yet.
keep_attrs (iterable) – A list, set, or tuple of the dotted namespace notations for attributes to keep within the object structure even if their values are None.

Return type:

TypeVar(T, bound= APIObject)

open_in_browser()

Opens class’ relevant web browser location. If default browser is not available the URL is logged.

Note: If text-mode browsers are used, the calling process will block until the user exits the browser.

Return type:: None

class datarobot.DatasetDetails

Represents a detailed view of a Dataset. The to_dataset method creates a Dataset from this details view.

Variables:

dataset_id (string) – The ID of this dataset.
name (string) – The name of this dataset in the catalog.
is_latest_version (bool) – Whether this dataset version is the latest version of this dataset.
version_id (string) – The object ID of the catalog_version the dataset belongs to.
categories (list(string)) – An array of strings describing the intended use of the dataset. The supported options are “TRAINING” and “PREDICTION”.
created_at (string) – The date when the dataset was created.
created_by (string) – Username of the user who created the dataset.
is_snapshot (bool) – Whether the dataset version is an immutable snapshot of data that was previously retrieved and saved to DataRobot.
data_persisted (Optional[bool]) – If true, user is allowed to view extended data profile (which includes data statistics like min/max/median/mean, histogram, etc.) and download data. If false, download is not allowed and only the data schema (feature names and types) will be available.
is_data_engine_eligible (bool) – Whether this dataset can be a data source of a data engine query.
processing_state (string) – Current ingestion process state of the dataset.
row_count (Optional[int]) – The number of rows in the dataset.
size (Optional[int]) – The size of the dataset as a CSV in bytes.
data_engine_query_id (string, optional) – ID of the source data engine query.
data_source_id (string, optional) – ID of the datasource used as the source of the dataset.
data_source_type (string) – The type of the datasource used as the source of the dataset.
description (string, optional) – The description of the dataset.
eda1_modification_date (string, optional) – The ISO 8601 formatted date and time when the EDA1 for the dataset was updated.
eda1_modifier_full_name (string, optional) – The user who last updated EDA1 for the dataset.
error (string) – Details of the exception raised during ingestion, if any.
feature_count (Optional[int]) – Total number of features in the dataset.
feature_count_by_type (list[FeatureTypeCount]) – Number of features in the dataset grouped by feature type.
last_modification_date (string) – The ISO 8601 formatted date and time when the dataset was last modified.
last_modifier_full_name (string) – Full name of the user who last modified the dataset.
tags (list[string]) – A list of tags attached to the item.
uri (string) – The URI to the data source, for example: - file_name.CSV - JDBC:DATA_SOURCE_GIVEN_NAME/SCHEMA.TABLE_NAME - JDBC:DATA_SOURCE_GIVEN_NAME/<query> - for query-based data sources - HTTPS://S3.amazonaws.com/my_data/my_dataset.CSV - etc.
sample_size (dict, optional) – The size of data fetched during dataset registration. For example, to fetch the first 95 rows, the sample_size value is {‘type’: ‘rows’, ‘value’: 95}. Currently only ‘rows’ type is supported.

classmethod get(dataset_id)

Gets the details for a Dataset from the server

Parameters:: dataset_id (str) – The ID for the Dataset from which to get details.
Return type:: DatasetDetails

to_dataset()

Build a Dataset object from the information in this object

Return type:: Dataset

class datarobot.models.dataset.ProjectLocation

ProjectLocation(url, id)

id: Alias for field number 1

url: Alias for field number 0

Secondary datasets

class datarobot.helpers.feature_discovery.SecondaryDataset

A secondary dataset to be used for feature discovery

Added in version v2.25.

Variables:

identifier (str) – Alias of the dataset (used directly as part of the generated feature names)
catalog_id (str) – Identifier of the catalog item
catalog_version_id (str) – Identifier of the catalog item version
snapshot_policy (Optional[str]) – Policy to use while creating a project or making predictions. If omitted, by default endpoint will use ‘latest’. Must be one of the following values: ‘specified’: Use specific snapshot specified by catalogVersionId ‘latest’: Use latest snapshot from the same catalog item ‘dynamic’: Get data from the source (only applicable for JDBC datasets)

Examples

import datarobot as dr
dataset_definition = dr.SecondaryDataset(
    identifier='profile',
    catalog_id='5ec4aec1f072bc028e3471ae',
    catalog_version_id='5ec4aec2f072bc028e3471b1',
)

Secondary dataset configurations

class datarobot.models.SecondaryDatasetConfigurations

Creates the secondary dataset configurations for a given project.

Added in version v2.20.

Variables:

id (str) – The ID of this secondary dataset configuration.
project_id (str) – The ID of the associated project.
config (list of DatasetConfiguration (Deprecated in version v2.23)) – A list of secondary dataset configurations.
secondary_datasets (list of SecondaryDataset (new in v2.23)) – A list of secondary datasets (secondaryDataset).
name (str) – The verbose name of the SecondaryDatasetConfig. null if it was not specified.
created (datetime.datetime) – A DataRobot-formatted datetime. null for legacy (before DR 6.0) database records.
creator_user_id (str) – The ID of the user who created this configuration.
creator_full_name (str) – The full name or email of the user who created this configuration.
featurelist_id (Optional[str]) – The ID of the feature list. null if it was not specified.
credential_ids (Optional[list of DatasetsCredentials]) – The credentials used by the secondary datasets if the datasets used in the configuration are from a data source.
is_default (Optional[bool]) – A bool flag indicating whether the default configuration was created during feature discovery AIM.
project_version (Optional[str]) – The version of the project when it was created (release version).

classmethod create(project_id, secondary_datasets, name, featurelist_id=None)

Create the secondary dataset configurations.

Added in version v2.20.

Parameters:

project_id (str) – The ID of the associated project.
secondary_datasets (list of SecondaryDataset (New in version v2.23)) – A list of secondary datasets used by the configuration. Each element is a datarobot.helpers.feature_discovery.SecondaryDataset.
name (str (New in version v2.23)) – The name of the secondary datasets configuration.
featurelist_id (str, or None (New in version v2.23)) – The ID of the feature list.

Return type:

An instance of SecondaryDatasetConfigurations.

Raises:

ClientError – Raised if incorrect configuration parameters are provided.

Examples

   profile_secondary_dataset = dr.SecondaryDataset(
       identifier='profile',
       catalog_id='5ec4aec1f072bc028e3471ae',
       catalog_version_id='5ec4aec2f072bc028e3471b1',
       snapshot_policy='latest'
   )

   transaction_secondary_dataset = dr.SecondaryDataset(
       identifier='transaction',
       catalog_id='5ec4aec268f0f30289a03901',
       catalog_version_id='5ec4aec268f0f30289a03900',
       snapshot_policy='latest'
   )

   secondary_datasets = [profile_secondary_dataset, transaction_secondary_dataset]
   new_secondary_dataset_config = dr.SecondaryDatasetConfigurations.create(
       project_id=project.id,
       name='My config',
       secondary_datasets=secondary_datasets
   )

>>> new_secondary_dataset_config.id
'5fd1e86c589238a4e635e93d'

delete()

Removes the secondary datasets configuration.

Added in version v2.21.

Raises:: ClientError – Raised if an invalid or already deleted secondary dataset configuration ID is provided.

Examples

# Deleting with a valid secondary_dataset_config id
status_code = dr.SecondaryDatasetConfigurations.delete(some_config_id)
status_code
>>> 204

Return type:: None

get()

Retrieve a single secondary dataset configuration for a given ID.

Added in version v2.21.

Returns:: secondary_dataset_configurations – The requested secondary dataset configuration.
Return type:: SecondaryDatasetConfigurations

Examples

config_id = '5fd1e86c589238a4e635e93d'
secondary_dataset_config = dr.SecondaryDatasetConfigurations(id=config_id).get()
>>> secondary_dataset_config
{
     'created': datetime.datetime(2020, 12, 9, 6, 16, 22, tzinfo=tzutc()),
     'creator_full_name': u'abc@datarobot.com',
     'creator_user_id': u'asdf4af1gf4bdsd2fba1de0a',
     'credential_ids': None,
     'featurelist_id': None,
     'id': u'5fd1e86c589238a4e635e93d',
     'is_default': True,
     'name': u'My config',
     'project_id': u'5fd06afce2456ec1e9d20457',
     'project_version': None,
     'secondary_datasets': [
            {
                'snapshot_policy': u'latest',
                'identifier': u'profile',
                'catalog_version_id': u'5fd06b4af24c641b68e4d88f',
                'catalog_id': u'5fd06b4af24c641b68e4d88e'
            },
            {
                'snapshot_policy': u'dynamic',
                'identifier': u'transaction',
                'catalog_version_id': u'5fd1e86c589238a4e635e98e',
                'catalog_id': u'5fd1e86c589238a4e635e98d'
            }
     ]
}

classmethod list(project_id, featurelist_id=None, limit=None, offset=None)

Returns a list of secondary dataset configurations.

Added in version v2.23.

Parameters:

project_id (str) – The ID of the project.
featurelist_id (Optional[str]) – The ID of the feature list to filter the secondary datasets configurations.

Returns:

secondary_dataset_configurations – The requested list of secondary dataset configurations for a given project.

Return type:

list of SecondaryDatasetConfigurations

Examples

pid = '5fd06afce2456ec1e9d20457'
secondary_dataset_configs = dr.SecondaryDatasetConfigurations.list(pid)
>>> secondary_dataset_configs[0]
    {
         'created': datetime.datetime(2020, 12, 9, 6, 16, 22, tzinfo=tzutc()),
         'creator_full_name': u'abc@datarobot.com',
         'creator_user_id': u'asdf4af1gf4bdsd2fba1de0a',
         'credential_ids': None,
         'featurelist_id': None,
         'id': u'5fd1e86c589238a4e635e93d',
         'is_default': True,
         'name': u'My config',
         'project_id': u'5fd06afce2456ec1e9d20457',
         'project_version': None,
         'secondary_datasets': [
                {
                    'snapshot_policy': u'latest',
                    'identifier': u'profile',
                    'catalog_version_id': u'5fd06b4af24c641b68e4d88f',
                    'catalog_id': u'5fd06b4af24c641b68e4d88e'
                },
                {
                    'snapshot_policy': u'dynamic',
                    'identifier': u'transaction',
                    'catalog_version_id': u'5fd1e86c589238a4e635e98e',
                    'catalog_id': u'5fd1e86c589238a4e635e98d'
                }
         ]
    }

Data engine query generator

class datarobot.DataEngineQueryGenerator

DataEngineQueryGenerator is used to set up time series data prep.

Added in version v2.27.

Variables:

id (str) – ID of the query generator.
query (str) – Text of the generated Spark SQL query.
datasets (list(QueryGeneratorDataset)) – Datasets associated with the query generator.
generator_settings (QueryGeneratorSettings) – The settings used to define the query.
generator_type (str) – TimeSeries is the only supported type.

classmethod create(generator_type, datasets, generator_settings)

Creates a query generator entity.

Added in version v2.27.

Parameters:

generator_type (str) – Type of data engine query generator.
datasets (List[QueryGeneratorDataset]) – Source datasets in the Data Engine workspace.
generator_settings (QueryGeneratorSettings) – Data engine generator settings of the given generator_type.

Returns:

query_generator – The created generator.

Return type:

DataEngineQueryGenerator

Examples

import datarobot as dr
from datarobot.models.data_engine_query_generator import (
   QueryGeneratorDataset,
   QueryGeneratorSettings,
)
dataset = QueryGeneratorDataset(
   alias='My_Awesome_Dataset_csv',
   dataset_id='61093144cabd630828bca321',
   dataset_version_id=1,
)
settings = QueryGeneratorSettings(
   datetime_partition_column='date',
   time_unit='DAY',
   time_step=1,
   default_numeric_aggregation_method='sum',
   default_categorical_aggregation_method='mostFrequent',
)
g = dr.DataEngineQueryGenerator.create(
   generator_type='TimeSeries',
   datasets=[dataset],
   generator_settings=settings,
)
g.id
>>>'54e639a18bd88f08078ca831'
g.generator_type
>>>'TimeSeries'

classmethod get(generator_id)

Returns the information about a query generator.

Parameters:: generator_id (str) – The identifier of the query generator you want to load.
Returns:: query_generator – The queried generator.
Return type:: DataEngineQueryGenerator

Examples

import datarobot as dr
g = dr.DataEngineQueryGenerator.get(generator_id='54e639a18bd88f08078ca831')
g.id
>>>'54e639a18bd88f08078ca831'
g.generator_type
>>>'TimeSeries'

create_dataset(dataset_id=None, dataset_version_id=None, max_wait=600)

A blocking call that creates a new Dataset from the query generator. Returns the Dataset after it has been successfully processed. If optional parameters are not specified the query is applied to the dataset_id and dataset_version_id stored in the query generator. If specified they will override the stored dataset_id/dataset_version_id, i.e., to prep a prediction dataset.

Parameters:

dataset_id (Optional[str]) – The ID of the unprepped dataset to apply the query to.
dataset_version_id (Optional[str]) – The version_id of the unprepped dataset to apply the query to.

Returns:

response – The Dataset created from the query generator.

Return type:

Dataset

prepare_prediction_dataset_from_catalog(project_id, dataset_id, dataset_version_id=None, max_wait=600, relax_known_in_advance_features_check=None)

Apply time series data prep to a catalog dataset and upload it to the project as a PredictionDataset.

Added in version v3.1.

Parameters:

project_id (str) – The ID of the project to which you upload the prediction dataset.
dataset_id (str) – The identifier of the dataset.
dataset_version_id (Optional[str]) – The version ID of the dataset to use.
max_wait (Optional[int]) – Optional, the maximum number of seconds to wait before giving up.
relax_known_in_advance_features_check (Optional[bool]) – For time series projects only. If True, missing values in the known in advance features are allowed in the forecast window at the prediction time. If omitted or False, missing values are not allowed.

Returns:

dataset – The newly uploaded dataset.

Return type:

PredictionDataset

prepare_prediction_dataset(sourcedata, project_id, max_wait=600, relax_known_in_advance_features_check=None)

Apply time series data prep and upload the PredictionDataset to the project.

Added in version v3.1.

Parameters:

sourcedata (str, file or pandas.DataFrame) – Data to be used for predictions. If it is a string, it can be either a path to a local file, or raw file content. If using a file on disk, the filename must consist of ASCII characters only.
project_id (str) – The ID of the project to which you upload the prediction dataset.
max_wait (Optional[int]) – The maximum number of seconds to wait for the uploaded dataset to be processed before raising an error.
relax_known_in_advance_features_check (Optional[bool]) – For time series projects only. If True, missing values in the known in advance features are allowed in the forecast window at the prediction time. If omitted or False, missing values are not allowed.

Returns:

dataset – The newly uploaded dataset.

Return type:

PredictionDataset

Raises:

InputNotUnderstoodError – Raised if sourcedata isn’t one of supported types.
AsyncFailureError – Raised if polling for the status of an async process resulted in a response with an unsupported status code.
AsyncProcessUnsuccessfulError – Raised if project creation was unsuccessful (i.e., the server reported an error in uploading the dataset).
AsyncTimeoutError – Raised if processing the uploaded dataset took more time than specified by the max_wait parameter.

Datasets

Secondary datasets

Secondary dataset configurations

Data engine query generator

Sharing access

Sharing role