**Use Spark ML**

**Overview**

BigDL provides `DLEstimator`

and `DLClassifier`

for users with Apache Spark MLlib experience, which
provides high level API for training a BigDL Model with the Apache Spark
Estimator/
Transfomer
pattern, thus users can conveniently fit BigDL into a ML pipeline. The fitted model `DLModel`

and
`DLClassiferModel`

contains the trained BigDL model and extends the Spark ML `Model`

class.
Alternatively users may also construct a `DLModel`

with a pre-trained BigDL model to use it in
Spark ML Pipeline for prediction. We are going to show you how to define a DLEstimator and
DLClassifier and how to use it. For advanced users, please check our
ML Pipeline API for detailed usage.

**Define a DLEstimator**

Before we are trying to use DLEstimator to automate the training process, we need to make clear which model used to be updated parameters and gradients, which criterion used to measure the loss, the dimension of the features and the label. These are the key elements the DLEstimator required to prepare for the training. If you are unfamiliar with creating a model and criterion, check out Model and Losses sections with provided links.

So, suppose we create a model with single linear layer and use MSECriterion as loss function here. You can choose any other model or criterion for your own good when you start your own training.

Then basically one can write code like this:

**Scala:**

```
val model = Sequential().add(Linear(2, 2))
val estimator = new DLEstimator(model, criterion, Array(2), Array(2))
```

**Python:**

```
linear_model = Sequential().add(Linear(2, 2))
mse_criterion = MSECriterion()
estimator = DLEstimator(model=linear_model, criterion=mse_criterion,
feature_size=[2], label_size=[2])
```

Now, you have a DLEstimator based on your own choice of model, criterion. Also, make sure your specified feature size and label size consistent with the actual ones otherwise exception will be encountered.

**Define a DLClassifier**

Since DLlassifier is the subclass of DLEstimator, the way of defining a DLClassifier is almost the same like creating a DLEstimator except that you don't need to specify the label size because it's set to default binary value and pay attention to choosing the criterion suitable for classification problem wisely.

Suppose we still create a model with single linear layer and use ClassNLL criterion as loss function here.

**Scala:**

```
val model = Sequential().add(Linear(2, 2)).add(LogSoftMax())
val criterion = ClassNLLCriterion()
val estimator = new DLClassifier(model, criterion, Array(2))
```

**Python:**

```
linear_model = Sequential().add(Linear(2, 2))
classNLL_criterion = ClassNLLCriterion()
classifier = DLClassifier(model=linear_model, criterion=classNLL_criterion,
feature_size=[2])
```

## Hyperparameter setting

Prior to the commencement of the training process, you can modify the batch size, the epoch number of your training, and learning rate to meet your goal or DLEstimator/DLClassifier will use the default value.

Continue the codes above, DLEstimator and DLClassifier can be setted in the same way.

**Scala:**

```
//for esitmator
estimator.setBatchSize(4).setMaxEpoch(10).setLearningRate(0.01)
//for classifier
classifier.setBatchSize(4).setMaxEpoch(10).setLearningRate(0.01)
```

**Python:**

```
# for esitmator
estimator.setBatchSize(4).setMaxEpoch(10).setLearningRate(0.01)
# for classifier
classifier.setBatchSize(4).setMaxEpoch(10).setLearningRate(0.01)
```

## Prepare the data and start the training process

Users need to convert the data into Spark's DataFrame/DataSet to feed to the DLEstimator/DLCLassifer. Then after these steps, we can start training now.

Suppose `df`

is the training data, simple call `fit`

method and let BigDL train the model for you. You will
get a DLModel or DLClassifierModel based on which one you choose from DLEstimator and DLClassifier.

**Scala:**

```
//get a DLModel
val dlModel = estimator.fit(df)
//get a DLClassifierModel
val dlClassifierModel = classifier.fit(df)
```

**Python:**

```
# get a DLModel
dlModel = estimator.fit(df)
# get a DLClassifierModel
dlClassifierModel = classifier.fit(df)
```

## Make prediction on chosen data by using DLModel/DLClassifierModel

Since DLModel/DLClassifierModel inherits from Spark's Transformer abstract class, simply call `transform`

method on DLModel/DLClassifierModel to make prediction.

**Scala:**

```
dlModel.transform(df).show(false)
```

**Python:**

```
dlModel.transform(df).show(false)
```