Skip to main content
Version: v3.0.0

Supported data sources

Snowflake Backend Removed

As of version 2.3, Snowflake as a storage backend is no longer supported. Snowflake data sources (SnowflakeTable and SnowflakeCursor) remain fully supported.

Data must first be ingested into Feature Store before it can be used. Ingesting is the act of uploading data into Feature Store.

Feature Store supports reading data from the following protocols:

  • s3 (internally reusing s3a client)
  • s3a
  • wasbs (encrypted) and wasb (legacy)
  • abfss (encrypted) and abfs (legacy)
  • http/https (data gets uploaded to internal storage)
  • drive (to read files from H2O Drive)
  • gs to read files from Google Cloud Storage
  • file (to upload files from the machine running the client)
note

Due to technical limitations of underlying libraries, reading from Google Cloud Storage isn't supported when FeatureStore is configured to utilize Google Cloud offline storage.

CSV​

CSV file format. Supported path locations are S3 bucket, Azure Blob Storage, HTTP/HTTPS URL, H20 Drive and the local file system.

User API:

Parameters:

  • path: str - path to csv file
  • delimiter: str - values delimiter
source = CSVFile(path=..., delimiter=...)

CSV folder​

CSV Folder source. Supported path locations are S3 bucket, Azure Blob Storage and the local file system.

User API:

Parameters:

  • root_folder: str - path to the root folder

  • delimiter: str - values delimiter

  • filter_pattern: str - Pattern to locate the files. To match the files at depth "N", the filter pattern must contain N expressions separated by "/" where each string is either an exact string or a regex pattern.

    • For example: filter_pattern="data/.*/.*/.*comp/.*" will match this file "data/1996-03-03/1/1679-comp/hello.json".
source = CSVFolder(root_folder=..., delimiter=..., filter_pattern=...)

Parquet​

Parquet file format. Supported path locations are S3 bucket, Azure Blob Storage, HTTP/HTTPS URL, H20 Drive and the local file system.

User API:

Parameters:

  • path: str - path to parquet file
source = ParquetFile(path=...)

Parquet folder​

Parquet folder source. Supported path locations are S3 bucket, Azure Blob Storage and the local file system.

User API:

Parameters:

  • root_folder: str - path to the root folder

  • filter_pattern: str - Pattern to locate the files. To match the files at depth "N", the filter pattern must contain N expressions separated by "/" where each string is either an exact string or a regex pattern.

    • For example: filter_pattern="data/.*/.*/.*comp/.*" will match this file "data/1996-03-03/1/1679-comp/hello.json".
source = ParquetFolder(root_folder=..., filter_pattern=...)

JSON​

JSON file format. Supported path locations are S3 bucket, Azure Blob Storage, HTTP/HTTPS URL, H20 Drive and the local file system. Different types of JSON formats are supported. Read more here to learn what types of JSON files are supported. By default multiline is set to False.

User API:

Parameters:

  • path: str - path to JSON file
  • multiline: bool - True whether the input is JSON where one entry is on multiple lines, otherwise False.
source = JSONFile(path=..., multiline=...)
note

Please keep in mind that a JSON object is an unordered set of name/value pairs. This means that using JSON files for extracting schema can produce a schema with a different order of features than that used in the file.

JSON folder​

JSON folder source. Supported path locations are S3 bucket, Azure Blob Storage and the local file system.

User API:

Parameters:

  • root_folder: str - path to the root folder

  • multiline: bool - True whether the input is JSON where one entry is on multiple lines, otherwise False.

  • filter_pattern: str - Pattern to locate the files. To match the files at depth "N", the filter pattern must contain N expressions separated by "/" where each string is either an exact string or a regex pattern.

    • For example: filter_pattern="data/.*/.*/.*comp/.*" will match this file "data/1996-03-03/1/1679-comp/hello.json".
source = JSONFolder(root_folder=..., multiline=..., filter_pattern=...)
note

Please keep in mind that a JSON object is an unordered set of name/value pairs. This means that using JSON files for extracting schema can produce a schema with a different order of features than that used in the file.

MongoDB​

Data stored in a MongoDb can be accessed by Feature Store as well. For a MongoDb authentication, environment variables

  • MONGODB_USER
  • MONGODB_PASSWORD will be used to provide user information.

User API:

Parameters:

  • connection_uri: str - a MongoDb server URI
    • E.g. connection_uri="mongodb+srv://my_cluster.mongodb.net/test"
  • database: str - Name of a database on the server
    • E.g. database="sample_guides"
  • collection: str - Name of a collection to read the data from
    • E.g. collection="planets"
source = MongoDbCollection(connection_uri=..., database= ..., collection = ...)

Delta table​

Delta table format. Table can be stored in either S3 or Azure Blob Storage.

User API:

Parameters:

  • path: str - path to delta table
  • version: int - (Optional) - version of the delta table
  • timestamp: str - (Optional) - timestamp of the data in the table
  • filter: DeltaTableFilter - (Optional) - Filter on the delta table
note

version and timestamp cannot be configured simultaneously; passing both raises ValueError.

# Time travel by version...
source = DeltaTable(path=..., version=..., filter=...)

# ...or by timestamp.
source = DeltaTable(path=..., timestamp=..., filter=...)

DeltaTableFilter API:

Parameters:

  • column: str - name of the column
  • operator: str - operator to be applied
  • value: str | int | float | bool - value to be applied on the filter
delta_filter = DeltaTableFilter(column=..., operator=..., value=...)

Supported operators​

The following are the supported operators: ==, !=, <, <=, >, and >=.

Valid parameter combinations​

  1. Path
  2. Path, Version
  3. Path, Version, Filter
  4. Path, Timestamp
  5. Path, Timestamp, Filter
  6. Path, Filter

JDBC​

JDBC table format. Currently, we support the following JDBC connections:

  • PostgreSQL
  • Teradata
  • Hive2 (with Kerberos authentication)

User API:

Parameters:

  • connection_url: str - connection string including the database name

  • table: str - table to fetch data from

  • query: str - query to fetch data from

  • partition_options: PartitionOptions - (Optional) parameters to enable parallel execution. These are applicable only when table is specified

    • PartitionOptions constitutes : num_partitions, partition_column, lower_bound, upper_bound, fetch_size
source = JdbcTable(connection_url=..., table=..., partition_options=PartitionOptions(num_partitions = ..., partition_column = ..., lower_bound = ..., upper_bound = ..., fetch_size=...))
source = JdbcTable(connection_url=..., query=...)

The format of the connection URL is a standard JDBC connection string, such as:

  • For Teradata, jdbc:teradata://host:port/database
  • For PostgreSQL, jdbc:postgresql://host:port/database
  • For Hive2, jdbc:hive2://host:port/database;principal=service/host@REALM

The database is a mandatory part of the connection string in the case of Feature Store. Note that only one of table or query is supported at the same time; passing both raises ValueError. Additionally, PartitionOptions can only be specified with table. These options must all be specified if any of them is specified. They describe how to partition the table when reading in parallel from multiple workers. partitionColumn must be a numeric, date, or timestamp column from the table in question. Notice that lowerBound and upperBound are just used to decide the partition stride, not for filtering the rows in table. All rows in the table will be partitioned and returned. This option applies only to reading.

Connecting to Hive2​

Hive2 connections require Kerberos authentication. The JDBC URL must include the HiveServer2 service principal. Pass your Kerberos principal and keytab file as KerberosCredentials when you call ingest(). See Credentials configuration for the full credentials reference.

from h2o_featurestore import JdbcTable, KerberosCredentials

source = JdbcTable(
    connection_url="jdbc:hive2://host:10000/default;principal=hive/host@REALM",
    table="transactions",
)

credentials = KerberosCredentials(
    principal="user@REALM",
    keytab_file="/path/to/user.keytab",
)

feature_set.ingest(source, credentials=credentials)

Snowflake table​

Extract data from Snowflake tables or queries.

User API:

Parameters:

Listed in constructor order:

  • url: str - url to Snowflake instance
  • warehouse: str - Snowflake warehouse
  • database: str - Snowflake database
  • schema: str - Snowflake schema
  • table: str - table to fetch data from
  • query: str - query to fetch data from
  • insecure: bool - if True, Snowflake will not perform SSL verification
  • proxy: Proxy - proxy specification
  • role: str - Snowflake role
  • account: str - Snowflake account name
note

table and query parameters cannot be configured simultaneously; passing both raises ValueError.

from h2o_featurestore import SnowflakeTable, Proxy
proxy = Proxy(host=..., port=..., user=..., password=...)

# Read a whole table...
source = SnowflakeTable(url=..., warehouse=..., database=..., schema=..., table=...,
                        insecure=..., proxy=proxy, role=..., account=...)

# ...or the result of a query.
source = SnowflakeTable(url=..., warehouse=..., database=..., schema=..., query=...,
                        insecure=..., proxy=proxy, role=..., account=...)
note

A proxy is an optional argument in the Snowflake data source API. If a proxy is not being used, the proxy configuration can simply be set to None.

The use of a proxy is possible for users only if the proxy feature is enabled by the administrator of the Snowflake account. Therefore, it is important to confirm whether proxy support is enabled before attempting to configure a proxy in the Snowflake data source API.

Snowflake Cursor object​

Extract data from Snowflake tables or queries.

User API:

The Snowflake Cursor object is currently only supported in the Python client.

Parameters:

Listed in constructor order:

  • url: str - url to Snowflake instance
  • warehouse: str - Snowflake warehouse
  • database: str - Snowflake database
  • schema: str - Snowflake schema
  • cursor: Any - an open Snowflake cursor object
  • insecure: bool - if True, Snowflake will not perform SSL verification
  • proxy: Proxy - proxy specification
  • role: str - Snowflake role
  • account: str - Snowflake account name
from h2o_featurestore import SnowflakeCursor

source = SnowflakeCursor(url=..., warehouse=..., database=..., schema=..., cursor=...,
                         insecure=..., proxy=..., role=..., account=...)

Database snippet:

Internally, the Snowflake Cursor is converted to SnowflakeTable with query and is therefore saved in the same format in the database.

Spark Data Frame​

When using Spark Data Frame as the source, several conditions must be met first. Read about the Spark dependencies to understand these requirements.

User API:

Parameters:

  • dataframe: DataFrame - Spark Data Frame instance
source = SparkDataFrame(dataframe=...)
note

A Spark data frame lives in the memory of the client's Spark session, so it is always uploaded from the client. It is written out as Parquet and uploaded through the same mechanism as local files, which means it also requires local data sources to be enabled by the administrator.

Uploading data from the local file system​

File based data sources can also point to a file or a folder on the machine running the client. Prefix the path with the file:// scheme and the client uploads the data into Feature Store before the job starts.

The following data sources support local paths:

Examples

csv = CSVFile(path="file:///home/user/data/transactions.csv", delimiter=",")
json = JSONFile(path="file:///home/user/data/transactions.json", multiline=True)
parquet_folder = ParquetFolder(root_folder="file:///home/user/data/transactions", filter_pattern="")

A path is treated as local only when it starts with file://. A bare path such as /home/user/data/transactions.csv is sent to Feature Store unchanged, and the job then fails because the server cannot see the client's disk.

The remaining data sources (Delta table, JDBC, Snowflake, MongoDB and BigQuery) do not support local paths. They describe data that Feature Store connects to directly.

How the upload works​

The client asks Feature Store for a set of time limited pre-signed upload URLs, uploads the files into Feature Store's temporary storage and then submits the job against that temporary location. Because Feature Store reads its own storage, no data source credentials are needed for local sources; any credentials passed alongside a local source are ignored.

Local sources can be used wherever a data source is accepted, except for scheduled ingestion, namely:

  • feature_set.ingest(source) and feature_set.ingest_async(source)
  • workspace.extract_schema_from_source(source) and workspace.extract_schema_from_source_async(source)
  • workspace.drafts.create(...) and workspace.drafts.create_major_version(...)
note

Local data sources must be enabled by the administrator of your Feature Store instance. When the feature is disabled, the upload fails with "Local data sources are disabled. Contact with administrator to enable this functionality".

warning

Local sources cannot be used for scheduled ingestion (feature_set.schedule_ingest) or lazy ingestion (feature_set.ingest_lazy). Those tasks run later on the server, when the client's files are no longer available, so passing a local source raises a ValueError. Upload the data to a permanent location such as S3 or Azure Blob Storage instead.

Uploading folders​

For the folder sources (CSVFolder, JSONFolder and ParquetFolder), the filter_pattern parameter decides which files are collected:

  • Without a filter_pattern, only the files directly inside the root folder are uploaded. The folder must not contain subdirectories in that case.
  • With a filter_pattern, the folder is traversed recursively and the files matching the pattern are uploaded, keeping their paths relative to the root folder.

Empty (zero byte) files are skipped. If a local file is empty, or if no non-empty file in a local folder matches, the client raises a ValueError.

Accessing H2O Drive Data​

When H2O Drive application is running in the same cloud environment as Feature Store, then user is able to access files that he/she uploaded into H2O Drive. To refer to those files, let's specify the scheme as drive.

Examples

source_1 = CSVFile("drive://example-file-1.csv")
source_2 = CSVFile("drive://my-subdirectory/example-file-2.csv")

BigQuery (Google Cloud)​

FeatureStore can extract data from BigQuery tables or queries.

User API:

Parameters:

  • table: str - table to fetch data from
  • parent_project: str - (Optional) The Google Cloud Project ID of the table to bill for the export
  • query: str - query to perform and read its result from
  • materialization_dataset: str - When a query parameter was specified, then a dataset where the materialized view is going to be created. This dataset should be in same location as the view or the queried tables.
note

table and query parameters cannot be configured simultaneously; passing both raises ValueError.

source = BigQueryTable(table=your_dataset.your_table, parent_project=your_project)

sql="SELECT label, count(1)  FROM `your_project.your_dataset.your_table` group by label"
source = BigQueryTable(parent_project=your_project, query=sql, materialization_dataset="your_temporal_dataset")

An instance of GcpCredentials will be utilized to authenticate to Google BigQuery. See info on GcpCredentials when a specific instance is required.

Feedback