Run on Google Cloud Dataproc

Deploy BigDL on Dataproc

Before using BigDL on Google Dataproc, you need setup a project and create a cluster on Dataproc(you may refer to for more instructions). Now you can create a Cloud Dataproc cluster using the Google Cloud SDK's( gcloud command-line tool.

To make it easy to try out BigDL on Spark on Dataproc, an initial action script is provided. You can use use this initialization action to create a new Dataproc cluster with BigDL pre-installed by You can use below command to create a new cluster with BigDL preinstall

gcloud dataproc clusters create <CLUSTER_NAME> --initialization-actions=gs://dataproc-initialization-actions/bigdl/


By default, it will automatically download BigDL 0.4.0 for Dataproc 1.2 (Spark 2.2.0 and Scala 2.11.8). To download a different version of BigDL or one targeted to a different version of Spark/Scala, find the download URL from the BigDL releases page, and set the metadata key "bigdl-download-url". eg.

gcloud dataproc clusters create <CLUSTER_NAME> \
    --image-version 1.0 \
    --initialization-actions gs://dataproc-initialization-actions/bigdl/ \
    --initialization-action-timeout 10m \
    --metadata 'bigdl-download-url='

More information please refer

Once the cluster is provisioned, you will be able to see the cluster running in the Google Cloud Platform Console. Now you can SSH to the master node.

Cloud Dataproc support various way to SSH to the master, here we use SSH from Google Cloud SDK. E.g.,

gcloud compute --project <PROJECT_ID> ssh --zone <ZONE> <CLUSTER_NAME>

Google cloud SDK will perform the authentication for you and open an SSH client (Eg Putty).

You should be able to find BigDL is located under /opt/intel-bigdl. And your VM is ready for running deep learning examples at scale!

Run BigDL Scala Examples

Now you can run BigDL examples on Google Dataproc. For instance, you may use the script which is located under ./bin directory with following parameters:

After the training, you can check the log files and generated models

Replace $BIGDLJAR with bigdl binary name in ./lib in below command, eg: bigdl-SPARK_2.2-0.3.0-jar-with-dependencies.jar

./bin/ --model lenet --nodes 2 --cores 2 --memory 1g --batch-size 16 -j lib/$BIGDLJAR -p spark_buildIn

You can also run lenet examples in below command. Before submit below command, please make sure you have already downloaded mnist and put it under mnist directory, more detail see

spark-submit \
 --executor-cores 2 \
 --num-executors 2 \
 --driver-class-path ./lib/$BIGDLJAR \
 --class \
 ./lib/$BIGDLJAR \
 -f ./mnist \
 -b 16

Run BigDL Python example

Download from


Replace $BIGDLJAR with bigdl binary name in ./lib, eg: bigdl-SPARK_2.2-0.3.0-jar-with-dependencies.jar
Replace $BIGDL_PYTHON_ZIP with bigdl python binary name in ./lib, eg:

spark-submit \
        --driver-cores 2  \
        --driver-memory 2g  \
        --num-executors 2  \
        --executor-cores 2  \
        --executor-memory 4g \
        --py-files ${PYTHON_API_ZIP_PATH},./  \
        --properties-file ./conf/spark-bigdl.conf \
        --jars ${BigDL_JAR_PATH} \
        --conf spark.driver.extraClassPath=${BigDL_JAR_PATH} \
        --conf spark.executor.extraClassPath=${BigDL_JAR_PATH} \
        ./ \
        --action train