Skip to main content

Using H2O-3 Secure on Spark (Sparkling Water)

H2O-3 Secure

Running on Spark through Sparkling Water is part of H2O-3 Secure: the supported, production-grade path for production deployments. Contact enterprise@h2o.ai.

Sparkling Water runs H2O-3 Secure machine learning algorithms on an Apache Spark cluster, so you can prepare data with Spark and train models with H2O-3 Secure. It converts data between Spark DataFrames, RDDs, and H2OFrames.

Sparkling Water provides a separate build for each supported Spark version, and builds are not interchangeable.

Requirements​

Before you start, make sure you have the following:

  • A Spark installation on your cluster.
  • A Spark version that Sparkling Water supports. For the current list, see the Sparkling Water documentation.

Download Sparkling Water​

To get the build that matches your version of Spark, contact enterprise@h2o.ai for the download link.

Start Sparkling Water​

After you download the build, start Sparkling Water and an H2O-3 Secure cluster inside Spark:

  1. Unzip the archive, set SPARK_HOME, then start the Sparkling Water shell:

    unzip sparkling-water-<version>.zip
    cd sparkling-water-<version>
    export SPARK_HOME=<path-to-spark>
    bin/sparkling-shell
  2. In the shell, start the H2O-3 Secure cluster:

    import ai.h2o.sparkling._
    val h2oContext = H2OContext.getOrCreate()

For configuration and deployment options, see the Sparkling Water documentation.

Language interfaces​

Sparkling Water provides Python and R interfaces in addition to Scala:

  • PySparkling is the Python interface. Contact enterprise@h2o.ai for the download link.
  • RSparkling is the R interface. It extends sparklyr. Use sparklyr to deploy and initialize Sparkling Water, then use the H2O-3 Secure R package to build models.

FAQ​

note

Some answers in this FAQ were written for earlier Sparkling Water releases. For the current API, see the Sparkling Water documentation.

What are the advantages of using Sparkling Water compared with standalone H2O-3 Secure?​

Sparkling Water contains the same features and functionality as H2O-3 Secure but provides a way to use H2O-3 Secure with Spark, a large-scale cluster framework.

Sparkling Water is ideal for H2O-3 Secure users who need to manage large clusters for their data processing needs and want to transfer data from Spark to H2O-3 Secure (or vice versa).

There is also a Python interface available to enable access to Sparkling Water directly from PySpark.

How do I change the memory size of the executors in a droplet?​

There are two ways to do this:

  • Change your default Spark setup in $SPARK_HOME/conf/spark-defaults.conf

or

  • Pass --conf via spark-submit when you launch your droplet. For example:

    $SPARK_HOME/bin/spark-submit --conf spark.executor.memory=4g --master $MASTER --class org.my.Droplet $TOPDIR/assembly/build/libs/droplet.jar

I received the following error while running Sparkling Water using multiple nodes, but not when using a single node. What should I do?​

onExCompletion for water.parser.ParseDataset$MultiFileParseTask@31cd4150
water.DException$DistributedException: from /10.23.36.177:54321; by class water.parser.ParseDataset$MultiFileParseTask; class water.DException$DistributedException: from /10.23.36.177:54325; by class water.parser.ParseDataset$MultiFileParseTask; class water.DException$DistributedException: from /10.23.36.178:54325; by class water.parser.ParseDataset$MultiFileParseTask$DistributedParse; class java.lang.NullPointerException: null
at water.persist.PersistManager.load(PersistManager.java:141)
at water.Value.loadPersist(Value.java:226)
at water.Value.memOrLoad(Value.java:123)
at water.Value.get(Value.java:137)
at water.fvec.Vec.chunkForChunkIdx(Vec.java:794)
at water.fvec.ByteVec.chunkForChunkIdx(ByteVec.java:18)
at water.fvec.ByteVec.chunkForChunkIdx(ByteVec.java:14)
at water.MRTask.compute2(MRTask.java:426)
at water.MRTask.compute2(MRTask.java:398)

This error output displays if the input file is not present on all nodes. Because of the way that Sparkling Water distributes data, the input file is required on all nodes (including remote), not just the primary node. Make sure there is a copy of the input file on all the nodes, then try again.

Are there any drawbacks to using Sparkling Water compared to standalone H2O-3 Secure?​

The version of H2O-3 Secure embedded in Sparkling Water is the same as the standalone version.

How do I use Sparkling Water from the Spark shell?​

Start the Sparkling Water shell from the Sparkling Water distribution:

bin/sparkling-shell

The software distribution provides example scripts in the examples/scripts directory:

bin/sparkling-shell -i examples/scripts/chicagoCrimeSmallShell.script.scala

Initialize H2O-3 Secure as shown in the following example:

import ai.h2o.sparkling._
val h2oContext = H2OContext.getOrCreate()

After successfully launching H2O-3 Secure, the following output displays:

Sparkling Water Context:
* number of executors: 3
* list of used executors:
(executorId, host, port)
------------------------
(1,Michals-MBP.0xdata.loc,54325)
(0,Michals-MBP.0xdata.loc,54321)
(2,Michals-MBP.0xdata.loc,54323)
------------------------

How do I use H2O-3 Secure with Spark Submit?​

Spark Submit is for submitting self-contained applications. For more information, refer to the Spark documentation.

First, initialize H2O-3 Secure:

import ai.h2o.sparkling._
val h2oContext = H2OContext.getOrCreate()

The Sparkling Water distribution provides several examples of self-contained applications built with Sparkling Water. To run the examples:

bin/run-example.sh ChicagoCrimeAppSmall

How do I use Sparkling Water with Databricks?​

Refer to the "Running Sparkling Water on Databricks Azure Cluster" in the Sparkling Water User Guide for information on how to use Sparkling Water with Databricks.

How do I develop applications with Sparkling Water?​

For a regular Spark application (a self-contained application as described in the Spark documentation), the app needs to initialize H2OContext:

import ai.h2o.sparkling._
val h2oContext = H2OContext.getOrCreate()

For more information, refer to the Sparkling Water Development documentation in the Sparkling Water User Guide.

How do I connect to Sparkling Water from R or Python?​

After you start H2OContext, point your client to the IP address and port number that H2OContext reports.

Additional resources​

Next steps​

To run H2O-3 Secure without Spark, see Downloading and installing and Starting H2O-3 Secure.


Feedback