After pip install


Precondition

Use an Interactive Shell

Run as a local program

# X_train, Y_train, X_test are all ndarray and the first dimension is the sample number.
local_optimizer = Optimizer.create(
    model=model_definition,
    training_set=(X_train, Y_train))
local_optimizer.predict(X_test)
local_optimizer.predict_class(X_test)

Use Jupyter Notebook

 jupyter notebook --notebook-dir=./ --ip=* --no-browser

Example code to verify if BigDL can run successfully

from bigdl.util.common import *
from pyspark import SparkContext
from bigdl.nn.layer import *
import bigdl.version

# create sparkcontext with bigdl configuration
sc = SparkContext.getOrCreate(conf=create_spark_conf().setMaster("local[*]"))
init_engine() # prepare the bigdl environment 
bigdl.version.__version__ # Get the current BigDL version
linear = Linear(2, 3) # Try to create a Linear layer

BigDL Configurations

export SPARK_DRIVER_MEMORY=20g

  Set the environment variables BIGDL_JARS and BIGDL_PACKAGES BEFORE creating SparkContext:

export BIGDL_JARS=...
export BIGDL_PACKAGES=...

  If you want to redirect spark logs to file and keep BigDL logs in console only, call the following API before you train your model:

from bigdl.util.common import *

# by default redirected to `bigdl.log` under the current workspace
redire_spark_logs(log_path="bigdl.log")
show_bigdl_info_logs()