Spark is a great way to make use of the available RAM on a Hadoop cluster to run fast in-memory analysis and queries, and H2O is a great project for running distributed machine learning algorithms on data stored in Hadoop. Together they form “Sparkling Water” (Spark + H2O, obviously!).
Easy to follow instructions for setting up Sparkling Water are available here: http://h2o-release.s3.amazonaws.com/sparkling-water/master/103/index.html
Running spark on Yarn is a good way to utilise an existing Hadoop cluster, however it’s challenging using the “live” method below to keep the Sparkling Water H2O Flow interface running permanently. Doing so can let a number of data scientists use the notebook style interface to run machine learning tasks. Luckily, using the spark-submit invocation with the water.SparklingWaterDriver class can ensure the web UI remains online even after the shell session which kicked it off exits (see below Persistent method).
Live method – doesn’t stay online after exiting shell session
- Create a shell script:
#!/bin/bash
export SPARK_HOME=’/usr/hdp/current/spark-client/’
export HADOOP_CONF_DIR=/etc/hadoop/conf
export MASTER=”yarn-client”
sparkling-water-1.3.5/bin/sparkling-shell –num-executors 3 –executor-memory 2g –master yarn-client - Run sparkling-shell
import org.apache.spark.h2o._
val h2oContext = new H2OContext(sc).start()
import h2oContext._
Persistent method – stays online even after exiting shell session
To start a “persistent” H2O cluster on Yarn (i.e. one which doesn’t exit immediately) simply run this command at the command line of a node where the spark client and sparkling water is installed:
nohup bin/spark-submit –class water.SparklingWaterDriver –master yarn-client –num-executors 3 –driver-memory 4g –executor-memory 2g –executor-cores 1 ../sparkling-water-0.2.1-58/assembly/build/libs/*.jar &
The Spark UI should be available on it’s usual port (http://XXX.XXX.XXX.XXX:54321) and should remain there even if the shell session which started the UI dies!
Thanks to the helpful and responsive folks at H2Oai for the above tip (originally answered here)!
Analytics Anvil 