Skip to main content
Version: v3.0.0

Supported derived transformation

Transformation changes the raw data and makes it usable by a model.

Spark pipeline​

Creating a feature set via Spark pipeline. The Spark pipeline generates the data from an existing feature set that you pass in as an input to the pipeline. Feature Store then uploads the Spark pipeline to the Feature Store artifacts cache and stores only the location of the pipeline in the database.

User API:

Parameters:

  • pipeline: str | pyspark.ml.Pipeline - you pass the local path to the pipeline (a .zip archive or a directory) or the pipeline object itself. A directory is archived, and a Pipeline object is first saved to a temporary directory and then archived; in both cases pipeline_local_location becomes the resulting pipeline.zip. When you pass a .zip path it is stored as-is. Once the feature set is registered, pipeline_remote_location holds the path to the uploaded Spark pipeline in the Feature Store artifacts cache.
note

Feature Store loads the artifact with Pipeline.load, so pass an unfitted pyspark.ml.Pipeline, or a path to one. A PipelineModel fails to load when the job runs, whether you pass the object or a path to a saved copy.

from h2o_featurestore import SparkPipeline
spark_pipeline_transformation = SparkPipeline("...")

Driverless AI MOJO​

Creating a feature set via Driverless AI MOJO. The MOJO pipeline generates the data from an existing feature set that you pass in as an input to the pipeline. Feature Store then uploads the MOJO pipeline to the Feature Store artifacts cache and stores only the location of the pipeline in the database.

note

Only features created from Driverless AI with the make_mojo_scoring_pipeline_for_features_only setting are supported in Feature Store.

User API:

Parameters:

  • mojo_local_location: str - you pass the local path to the MOJO pipeline. Once the feature set is registered, the transformation's mojo_remote_location attribute holds the path to the uploaded MOJO pipeline in the Feature Store artifacts cache.
  • shapley_value_type: ShapleyValueType - the Shapley value computation to perform. Defaults to ShapleyValueType.NONE.

ShapleyValueType

The following values are available:

  • ShapleyValueType.NONE—Feature Store does not compute Shapley values. This is the default.
  • ShapleyValueType.ORIGINAL—Feature Store computes Shapley values for the original input features. Use this to understand which raw inputs drive the model's prediction.
  • ShapleyValueType.TRANSFORMED—Feature Store computes Shapley values for the transformed features. Use this to understand the model's internally engineered representation.
from h2o_featurestore import DriverlessAIMOJO
from h2o_featurestore.core.transformations import ShapleyValueType

transformation = DriverlessAIMOJO("...", shapley_value_type=ShapleyValueType.TRANSFORMED)

# To use original input feature contributions instead:
# transformation = DriverlessAIMOJO("...", shapley_value_type=ShapleyValueType.ORIGINAL)
Performance impact

When you enable Shapley values, the MOJO pipeline runs twice—once for base predictions and once for Shapley contributions. This may significantly increase the time to generate predictions during ingestion (roughly doubling in many cases).

  • The MOJO must support Shapley contributions for the requested type; otherwise, Feature Store raises an error during ingestion.
  • Feature Store appends Shapley contribution columns after the base prediction columns in the output. The Driverless AI MOJO pipeline determines column names, which vary by model. Shapley columns are included when you retrieve the feature set.
  • The default value is ShapleyValueType.NONE, so existing MOJO transformations are unaffected.

JoinFeatureSets​

Creating a new feature set by joining together two different feature sets.

User API:

Parameters:

  • left_key: str - joining key which must be present in left feature set. Required, but ignored when join_type is JoinFeatureSetsType.CROSS.
  • right_key: str - joining key which must be present in right feature set. Required, but ignored when join_type is JoinFeatureSetsType.CROSS.
  • join_type: JoinFeatureSetsType - the join type. Defaults to JoinFeatureSetsType.INNER.
  • left_features: list[str] - optional subset of features to keep from the left feature set. When empty, all features are kept. In a keyed join left_key is always retained, even if you omit it from this list.
  • right_features: list[str] - optional subset of features to keep from the right feature set. When empty, all features are kept. In a keyed join right_key is always retained, even if you omit it from this list.

Feature names are matched case-insensitively, and listing a key in its own *_features list does not duplicate it in the output.

Output column names

Non-key columns from both sides are prefixed with their feature set name, as <feature_set_name>_<column>. This applies to every join type. Key columns are exempt from the prefix, except where a full join keeps two same-named keys and they would otherwise collide.

The key columns themselves depend on how the two keys relate:

  • Same name - an inner, left, or right join emits a single key column under that name. A full join keeps both, prefixed.
  • Different names - both columns appear under their own names, in every join type including full.

Cross joins

JoinFeatureSetsType.CROSS produces a Cartesian product, so there is no key to retain and left_key / right_key are ignored. left_features and right_features still apply — when empty, all features are kept. Since there is no key, every output column is prefixed.

JoinFeatureSetsType

The following values are available:

  • JoinFeatureSetsType.INNER—The inner join is the default join in Spark SQL. It selects rows that have matching values in both relations.
  • JoinFeatureSetsType.LEFT—A left join returns all values from the left relation and the matched values from the right relation, or appends NULL if there is no match.
  • JoinFeatureSetsType.RIGHT—A right join returns all values from the right relation and the matched values from the left relation, or appends NULL if there is no match.
  • JoinFeatureSetsType.FULL—A full join returns all values from both relations, appending NULL values on the side that does not have a match.
  • JoinFeatureSetsType.CROSS—A cross join returns the Cartesian product of two relations.
from h2o_featurestore import JoinFeatureSets, JoinFeatureSetsType

transformation = JoinFeatureSets(
    left_key="customer_id",
    right_key="customer_id",
    join_type=JoinFeatureSetsType.LEFT,
    left_features=["age", "income"],
)
note

When you do not pass join_type, Feature Store performs an inner join.

Feedback