Feature set API

Registering a feature set

To register a feature set, you first need to obtain the schema. See Schema API for information on how to create the schema.

Python

workspace = client.workspaces.list(name="my_workspace")[0]
workspace.feature_sets.register(schema, "feature_set_name", description="", primary_key=None, time_travel_column=None, time_travel_column_format="yyyy-MM-dd HH:mm:ss", partition_by=None, time_travel_column_as_partition=False)

If the partition_by argument is not set, the time travel column will be used by Feature Store to partition the layout by each ingestion. If it is defined, time_travel_column_as_partition can be set to True to use time travel based partitioning additionally.

note

In case primary key or partition by arguments contain same feature multiple times, only distinct values are used.

note

If value in primary key or partition by or time travel column corresponds to two or more features, most nested is selected by default. In other cases, specific feature can be selected by enclosing the feature name in ``

For example, feature set contains feature named "test.data" and second feature "test" with nested feature "data". But default for value "test.data", nested feature "data" will be selected. If feature with name "test.data" should be selected, value should be changed to "`.test.data`"

note

Feature Store is using time format used by Spark. Specification is available here.

note

If users wants to create feature sets which are accessible only by the owner and users the owner gave permission to, that feature set should be created in a private workspace.

To see naming conventions for feature set names, please visit Default naming rules.

To register a derived feature set, you first need to obtain the derived schema. See Schema API for information on how to create the schema.

Python

from h2o_featurestore import SparkPipeline
spark_pipeline_transformation = SparkPipeline("...")

workspace = client.workspaces.list(name="my_workspace")[0]
derived_schema = workspace.extract_derived_schema([parent_feature_set], spark_pipeline_transformation)

workspace.feature_sets.register(derived_schema, "derived_feature_set", description="", primary_key=None, time_travel_column=None, time_travel_column_format="yyyy-MM-dd HH:mm:ss", partition_by=None, time_travel_column_as_partition=False)

Features can be masked by setting Special Data fields in the schema. For further information, please visit Modify special data on a schema.

Setting any of the following attributes to true marks the feature for masking:

• spi - Sensitive Personal Information
• pci - Payment Card Industry
• rpi - Real Property Inventory
• demographic
• sensitive

Any of the special data tags would allow for the masking functionality to work and separate sensitive consumer output (e.g. unmasked data) from the masked view that the consumer role sees. Which tag is selected is more bookkeeping than leading to different functionality.

note

Feature Store does not support registering feature sets with the following characters in column names:

• ,
• ;
• { or }
• ( or )
• new line character
• tab character
• =

Time travel column selection

You can specify a time travel column during the registration call. If the column is specified, Feature Store will use that column to obtain time travel data and will use it for incremental ingest purposes. The explicitly passed time travel column must be present in the schema passed to the registration call.

If the time travel column is not specified, a virtual one is created, so you can still do time travel on static feature sets. Each ingestion to this feature set is treated as a new batch of data with a new timestamp.

Use the following register method argument to specify the name of the time travel column explicitly:

Python

time_travel_column

Inferring the data type of date-time columns during feature set registration

File types without schema information: For file types that have no metadata about column types (e.g., CSV), Feature Store detects date-time columns as regular string.

File types containing schema information: For file types that keep information about the data types (e.g., Parquet), Feature Store respects those types. If a date-time column is stored with a type of Timestamp or Date, Feature Store will respect that during the registration.

Listing feature sets within a workspace

note

The list method does not return feature sets directly. Instead, it returns an iterator which obtains the feature sets lazily.

Python

workspace = client.workspaces.list(name="my_workspace")[0]
workspace.feature_sets.list(query=None, advanced_search_options=None)

The query and advancedSearchOption arguments are optional and specify which feature sets should be returned. By default, no filtering options are specified.

To filter feature sets by name, description or tags please use query parameter.

Python

workspace.feature_sets.list(query="My feature")

The advancedSearchOption allows to filter feature sets by feature name, description or tags.

To provide the 'advancedSearchOption' in your requests, follow these steps:

Python

from h2o_featurestore.core.search_operator import SearchOperator
from h2o_featurestore.core.search_field import SearchField
from h2o_featurestore import AdvancedSearchOption
search_options = [AdvancedSearchOption(search_operator=SearchOperator.SEARCH_OPERATOR_LIKE, search_field=SearchField.SEARCH_FIELD_FEATURE_NAME, search_value="super feature")]

workspace.feature_sets.list(advanced_search_options=search_options)

Both parameters could be used together.

You can also list all major versions of the feature set:

Python

fs.major_versions()

This call shows all major versions of the feature set (the current and previous ones).

You can also list all versions of the feature set:

Python

fs.list_versions()

This call shows all versions of the feature set (the current and previous ones).

Listing feature sets across workspaces

workspace.feature_sets.list() is scoped to a single workspace. To search several workspaces at once, or every workspace you can access, use client.workspaces.list_feature_sets().

Python

# List feature sets across specific workspaces
for fs in client.workspaces.list_feature_sets(
    workspace_names=["workspaces/<uid_A>", "workspaces/<uid_B>"]
):
    print(fs.name)

# List across all accessible workspaces
for fs in client.workspaces.list_feature_sets():
    print(fs.name)

Workspaces are identified by their resource name (workspaces/<uid>), not their display name. You can construct this from a workspace object: f"workspaces/{workspace.uid}". Like the workspace-scoped list(), this method returns a lazy iterator (i.e., you can iterate over them without loading everything into memory at once). For details, see List feature sets across workspaces.

Feature sets found this way can be used as parents of a derived feature set even when they live in another workspace — see Joining feature sets from different workspaces.

Obtaining a feature set

Python

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("feature_set_name", version=None)

If the version is not specified, the latest version of the feature set is returned.

It is also possible to obtain different version of a feature set from some feature set instance as:

Python

fs = feature_set.get_version("2.1")

Commonly used properties

The following table lists the most commonly accessed properties. For the full set of updatable fields, see Updating feature set fields.

Property	Type	Description
parent	str	Workspace resource name that owns this feature set (e.g. workspaces/<uid>).
name	str	Display name of the feature set.
version	str	Current version (e.g. 1.0).
description	str	Description.
tags	list[str]	User-defined tags.
primary_key	list[str]	Primary key column names.
time_travel_column	str | None	Name of the time travel column, or None if one was not specified at registration (Feature Store creates a virtual one in that case).
deprecated	bool	Whether the feature set is deprecated.
features	dict	Map of feature name to Feature object.

Previewing data

You can preview up to a maximum of 100 rows and 50 features.

Python

fs.get_preview()

Setting feature set permissions

Refer to Permissions for more information.

Deleting feature sets

Python

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("name")
fs.delete()

Deleting feature set major versions

Python

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("name")
fs.delete_version(major_version=2)

Updating feature set fields

To update feature set fields, call the update() method with the fields you want to change. Only the fields you pass are modified, for example:

Python

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("name")

# Update one or more fields via update()
fs.update(deprecated=True)
# Overwrite the tags (tags accepts a list of strings)
fs.update(tags=["new tag 1", "new tag 2"])
# Overwrite the data source domains (accepts a list of strings)
fs.update(data_source_domains=["new domain 1", "new domain 2"])

# Time to live is set through its own setters
fs.time_to_live.ttl_offline = 2

Feature type can be changed by:

Python

from h2o_featurestore.core.resources.feature import FeatureType
workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("name")
feature = fs.features["feature"]
feature.profile.feature_type = FeatureType.CATEGORICAL

The following fields can be updated via fs.update(...) / feature.update(...):

Python

# fs.update(...)
- tags
- data_source_domains
- feature_set_type
- description
- application_name
- application_id
- deprecated
- custom_data
- state
- name

# feature.update(...), where feature = fs.features["<name>"]
- status
- importance
- description
- special
- classifiers
- anomaly_detection

The remaining fields are not update() kwargs — they are exposed as properties with their own setters, which apply the change immediately:

- fs.time_to_live.ttl_offline
- fs.time_to_live.ttl_offline_interval
- fs.time_to_live.ttl_online
- fs.time_to_live.ttl_online_interval
- feature.profile.feature_type

note

• feature_set_type has two values, RAW or ENGINEERED. It denotes whether the feature set was derived from raw or processed data. This classification exists for information purposes and does not affect Feature Store behavior.
• time_to_live is currently respected for data in online feature store only. It indicates the duration for which records remain stored before they are evicted.

To retrospectively find out who and when updated a feature set, call:

Python

fs.last_updated_by
fs.last_modified_time

Recommendation and classifiers

Refer to the Recommendation API for more information.

New version API

Refer to the Create new feature set version API for more information.

Feature set schema API

Getting schema

To get feature set's schema, run:

Python

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("gdp")
fs.schema.get()

Checking schema compatibility

To compare feature set's schema with the new data source's schema, run:

Python

from h2o_featurestore import CSVFile

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("gdp")
source = CSVFile("<path to csv file>")
new_schema = workspace.extract_schema_from_source(source)
fs.schema.is_compatible_with(new_schema, compare_data_types=True)

Parameters explanation:

Python

• new_schema new schema to check compatibility with.

• compare_data_types accepts True/False, indicates whether data type needs to be compared or not.

  • If compare_data_types is True, then data types for features with same name will be verified.
  • If compare_data_types is False, then data types for features with same name will not be verified.

Patching new schema

Patch schema checks for matching features between the 'new schema' and the existing 'fs.schema'. If there is a match, then the meta data such as special_data, description etc are copied into the new_schema

To patch the new schema with feature set's schema, run:

Python

from h2o_featurestore import CSVFile

workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("gdp")
source = CSVFile("<path to csv file>")
new_schema = workspace.extract_schema_from_source(source)
fs.schema.patch_from(new_schema, compare_data_types=True)

Parameters explanation:

Python

• new_schema new schema that needs to be patched.

• compare_data_types accepts True/False, indicates whether data type are to be compared while patching.

  • If compare_data_types is True, then data type from feature set schema is retained for features with same name and different types.
  • If compare_data_types is False, then data type from new schema is retained for features with same name and different types.

Offline to online API

To push existing data from offline Feature store into online, run:

Blocking approach:

Python

feature_set.materialize_online()

Non-Blocking approach:

Python

future = feature_set.materialize_online_async()

note

Feature set must have a primary key and time travel column defined in order to materialize the offline store into online.

More information about asynchronous methods is available at Asynchronous methods.

Subsequent calls to materialization only push the new records since the last call to online.

Online to offline API

There is a background process that periodically starts online to offline ingestion, but in case there is a need to push existing data from online Feature store into offline earlier than scheduled, then run:

Blocking approach:

Python

feature_set.start_online_offline_ingestion()

Non-Blocking approach:

Python

job = feature_set.start_online_offline_ingestion_async()

Feature set jobs API

You can get the list of jobs that are currently processing for the specific feature set by running:

Python

You can also retrieve a specific type of job by specifying the job_type parameter.

from h2o_featurestore import JobType

fs.get_active_jobs()
fs.get_active_jobs(JobType.INGEST)

Available JobType values:

• JobType.BACKFILL
• JobType.BATCH_RETRIEVE
• JobType.COMPUTE_RECOMMENDATION_CLASSIFIERS
• JobType.COMPUTE_STATISTICS
• JobType.EXTRACT_SCHEMA
• JobType.INGEST
• JobType.MATERIALIZATION_ONLINE
• JobType.OPTIMIZE_STORAGE
• JobType.RETRIEVE
• JobType.REVERT_INGEST
• JobType.UNKNOWN

Refreshing feature set

To refresh the feature set to contain the latest information, call:

Python

fs.refresh()

Getting recommendations

To get recommendations, call:

Python

fs.get_recommendations()

The following conditions must hold for recommendations:

• The feature set must have at least one or more classifiers defined.
• The results will be based on the retrieve permissions of the user.

Marking feature as target variable

When feature sets are used to train ML models, it can be beneficial to know which feature was used as model's target variable. In order to communicate this knowledge between different feature set users, there is a possibility to mark/discard a feature as a target variable and list those marked features.

Python

feature_state = fs.features["state"]
feature_state.mark_as_target_variable()

fs.list_features_used_as_target_variable()

feature_state.discard_as_target_variable()

Listing feature set users

From feature set owner's perspective, it may be needed to understand who is actually allowed to access and modify the given feature set. Therefore, there are convenience methods to list feature set users according to their rights. Each of these methods returns an iterator of users that have specified or higher rights. Each returned user also exposes access_type and resource_type (workspace or feature set), specifying where the access right permission comes from. See Permissions for the full add_*/remove_*/list_* contract.

note

The list method does not return users directly. Instead, it returns an iterator which obtains the users lazily.

Python

# listing users by access rights
workspace = client.workspaces.list(name="my_workspace")[0]
fs = workspace.feature_sets.get_by_name("training_fs")
owners = fs.list_owners()
editors = fs.list_editors()
sensitive_consumers = fs.list_sensitive_consumers()
consumers = fs.list_consumers()
viewers = fs.list_viewers()
metadata_viewers = fs.list_metadata_viewers()

# accessing returned element
owner = next(owners)
owner.email
owner.access_type
owner.resource_type

Artifacts

Refer to the Artifacts API for more information.

Derived feature sets

As mentioned in the beginning, a (derived) feature set can be defined in terms of other features sets and a transformation. There are several convenience methods that help you find out a lineage of a given feature set.

note

• In the Feature Store, lineage is preserved by tracking the ingest history. This allows users to identify the data source from which the ingest occurred.
• Users can create derived feature sets which are transformations of existing feature sets. This relationship is also preserved within the Feature Store.

Is the feature set a derived one or not?

Python

fs.is_derived()

Which feature sets were used to define this derived feature set?

Python

parent_feature_sets = fs.get_parent_feature_sets()

To get a list of derived feature set(s) that were build upon this feature set.

Python

derived_feature_sets = fs.get_derived_feature_sets()

Open feature set in Web UI

This method opens the given feature set in Feature Store Web UI.

Python

fs.open_website()

Optimizing feature set storage (Delta lake backend only)

In special cases, there can be a performance benefit when a feature set's data gets optimized. In order to manually enforce a storage optimization use following call. By default, feature set storage gets optimized by Z-order optimization for primary key(s). In case an optimization for different feature's list is needed, you can specify the optimization explicitly when making the call.

The optimization call returns optimization metrics provided by storage. Furthermore, a new minor feature set version gets created. The updated feature set version contains optimization input as one of its attributes.

Python

from h2o_featurestore import ZOrderByOptimization

# z-order optimization for primary key(s) by default
metrics = fs.optimize_storage()

# optimize_storage returns the optimization metrics string directly
print(metrics)

# z-order optimization for specific columns
fs.optimize_storage(ZOrderByOptimization(["name", "age"]))

# refresh version and show optimization input
fs.refresh()
fs.storage_optimization

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O Feature Store to cloud-feedback@h2o.ai