Initalizers
Zeros
Scala:
val initMethod = Zeros
Python:
init_method = Zeros()
Initialization method that set tensor to zeros.
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = Zeros
val biasInitMethod = Zeros
val model = Linear(3, 2).setName("linear1")
model.setInitMethod(weightInitMethod, biasInitMethod)
println(model.getParametersTable().get("linear1").get)
{
weight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
bias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradBias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradWeight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
}
Python example:
from bigdl.nn.initialization_method import *
weight_init = Zeros()
bias_init = Zeros()
model = Linear(3, 2)
model.set_init_method(weight_init, bias_init)
print("weight:")
print(model.get_weights()[0])
print("bias: ")
print(model.get_weights()[1])
creating: createZeros
creating: createZeros
creating: createLinear
weight:
[[ 0. 0. 0.]
[ 0. 0. 0.]]
bias:
[ 0. 0.]
Ones
Scala:
val initMethod = Ones
Python:
init_method = Ones()
Initialization method that set tensor to be ones.
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = Ones
val biasInitMethod = Ones
val model = Linear(3, 2).setName("linear1")
model.setInitMethod(weightInitMethod, biasInitMethod)
println(model.getParametersTable().get("linear1").get)
{
weight: 1.0 1.0 1.0
1.0 1.0 1.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
bias: 1.0
1.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradBias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradWeight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
}
Python example:
from bigdl.nn.initialization_method import *
weight_init = Ones()
bias_init = Ones()
model = Linear(3, 2)
model.set_init_method(weight_init, bias_init)
print("weight:")
print(model.get_weights()[0])
print("bias: ")
print(model.get_weights()[1])
creating: createOnes
creating: createOnes
creating: createLinear
weight:
[[ 1. 1. 1.]
[ 1. 1. 1.]]
bias:
[ 1. 1.]
ConstInitMethod
Scala:
val initMethod = ConstInitMethod(value: Double)
Python:
init_method = ConstInitMethod(value)
Initialization method that set tensor to the specified constant value.
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = ConstInitMethod(0.2)
val biasInitMethod = ConstInitMethod(0.2)
val linear = Linear(3, 2).setName("linear1")
linear.setInitMethod(weightInitMethod, biasInitMethod)
println(linear.getParametersTable().get("linear1").get)
{
weight: 0.2 0.2 0.2
0.2 0.2 0.2
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
bias: 0.2
0.2
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradBias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradWeight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
}
Python example:
from bigdl.nn.initialization_method import *
weight_init = ConstInitMethod(0.2)
bias_init = ConstInitMethod(0.2)
linear = Linear(3, 2)
linear.set_init_method(weight_init, bias_init)
print("weight:")
print(linear.get_weights()[0])
print("bias: ")
print(linear.get_weights()[1])
creating: createConstInitMethod
creating: createConstInitMethod
creating: createLinear
weight:
[[ 0.2 0.2 0.2]
[ 0.2 0.2 0.2]]
bias:
[ 0.2 0.2]
Xavier
Scala:
val initMethod = Xavier
Python:
init_method = Xavier()
The Xavier initialization method draws samples from a uniform distribution bounded by [-limit, limit) where limit = sqrt(6.0/(fanIn+fanOut)). The rationale behind this formula can be found in the paper Understanding the difficulty of training deep feedforward neural networks.
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = Xavier
val biasInitMethod = Xavier
val model = Linear(3, 2).setName("linear1")
model.setInitMethod(weightInitMethod, biasInitMethod)
println(model.getParametersTable().get("linear1").get)
{
weight: -0.78095555 -0.09939616 0.12034761
-0.3019594 0.11734331 0.80369484
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
bias: 1.0727772
-0.6703765
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradBias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradWeight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
}
Python example:
from bigdl.nn.initialization_method import *
weight_init = Xavier()
bias_init = Xavier()
model = Linear(3, 2)
model.set_init_method(weight_init, bias_init)
print("weight:")
print(model.get_weights()[0])
print("bias: ")
print(model.get_weights()[1])
creating: createXavier
creating: createXavier
creating: createLinear
weight:
[[ 0.00580597 -0.73662472 0.13767919]
[ 0.16802482 -0.49394709 -0.74967551]]
bias:
[-1.12355328 0.0779365 ]
BilinearFiller
Scala:
val initMethod = BilinearFiller
Python:
init_method = BilinearFiller()
Initialize the weight with coefficients for bilinear interpolation. A common use case is with the DeconvolutionLayer acting as upsampling. This initialization method can only be used in the weight initialization of SpatialFullConvolution.
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = BilinearFiller
val biasInitMethod - Zeros
val model = SpatialFullConvolution(2, 3, 2, 2).setName("sfconv")
model.setInitMethod(weightInitMethod, biasInitMethod)
println(model.getParametersTable().get("sfconv").get)
{
weight: (1,1,1,.,.) =
1.0 0.0
0.0 0.0
(1,1,2,.,.) =
1.0 0.0
0.0 0.0
(1,1,3,.,.) =
1.0 0.0
0.0 0.0
(1,2,1,.,.) =
1.0 0.0
0.0 0.0
(1,2,2,.,.) =
1.0 0.0
0.0 0.0
(1,2,3,.,.) =
1.0 0.0
0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 1x2x3x2x2]
bias: 0.0
0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 3]
gradBias: 0.0
0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 3]
gradWeight: (1,1,1,.,.) =
0.0 0.0
0.0 0.0
(1,1,2,.,.) =
0.0 0.0
0.0 0.0
(1,1,3,.,.) =
0.0 0.0
0.0 0.0
(1,2,1,.,.) =
0.0 0.0
0.0 0.0
(1,2,2,.,.) =
0.0 0.0
0.0 0.0
(1,2,3,.,.) =
0.0 0.0
0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 1x2x3x2x2]
}
Python example:
from bigdl.nn.initialization_method import *
weight_init = BilinearFiller()
bias_init = Zeros()
model = SpatialFullConvolution(2, 3, 2, 2)
model.set_init_method(weight_init, bias_init)
print("weight:")
print(model.get_weights()[0])
print("bias: ")
print(model.get_weights()[1])
creating: createBilinearFiller
creating: createZeros
creating: createSpatialFullConvolution
weight:
[[[[[ 1. 0.]
[ 0. 0.]]
[[ 1. 0.]
[ 0. 0.]]
[[ 1. 0.]
[ 0. 0.]]]
[[[ 1. 0.]
[ 0. 0.]]
[[ 1. 0.]
[ 0. 0.]]
[[ 1. 0.]
[ 0. 0.]]]]]
bias:
[ 0. 0. 0.]
RandomNormal
Scala:
val initMethod = RandomNormal(mean, stdv)
Python:
init_method = RandomNormal(mean, stdv)
This initialization method draws samples from a normal distribution.
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = RandomNormal(0, 1)
val biasInitMethod = RandomNormal(0, 1)
val linear = Linear(3, 2).setName("linear1")
linear.setInitMethod(weightInitMethod, biasInitMethod)
println(linear.getParametersTable().get("linear1").get)
{
weight: -0.5908564 0.32844943 -0.845019
0.21550806 1.2037253 0.6807024
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
bias: 0.5345903
-0.76420456
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradBias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradWeight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
}
Python example:
from bigdl.nn.initialization_method import *
from bigdl.nn.layer import *
weight_init = RandomNormal(0, 1)
bias_init = RandomNormal(0, 1)
linear= Linear(3, 2)
linear.set_init_method(weight_init, bias_init)
print("weight:")
print(linear.get_weights()[0])
print("bias: ")
print(linear.get_weights()[1])
creating: createRandomNormal
creating: createRandomNormal
creating: createLinear
weight:
[[-0.00784962 0.77845585 -1.16250944]
[ 0.03195094 -0.15211993 0.6254822 ]]
bias:
[-0.37883148 -0.81106091]
RandomUniform
Scala:
val initMethod = RandomUniform(lower, upper)
Python:
init_method = RandomUniform(upper=None, lower=None)
This initialization method draws samples from a uniform distribution. If the lower bound and upper bound of this uniform distribution is not specified, it will be set to [-limit, limit) where limit = 1/sqrt(fanIn).
Scala example:
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
val weightInitMethod = RandomUniform
val biasInitMethod = RandomUniform(0, 1)
val model = Linear(3, 2).setName("linear1")
model.setInitMethod(weightInitMethod, biasInitMethod)
println(model.getParametersTable().get("linear1").get)
{
weight: -0.572536 0.13046022 -0.040449623
-0.547542 0.19093458 0.5632484
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
bias: 0.785292
0.63280666
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradBias: 0.0
0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
gradWeight: 0.0 0.0 0.0
0.0 0.0 0.0
[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x3]
}
Python example:
from bigdl.nn.initialization_method import *
weight_init = RandomUniform()
bias_init = RandomUniform()
model = Linear(3, 2)
model.set_init_method(weight_init, bias_init)
print("weight:")
print(model.get_weights()[0])
print("bias: ")
print(model.get_weights()[1])
creating: createRandomUniform
creating: createRandomUniform
creating: createLinear
weight:
[[ 0.53153235 0.53016287 0.32831791]
[-0.45736417 -0.16206641 0.21758588]]
bias:
[ 0.32058391 0.26307678]
Define your own Initializer
All customizedInitializer should implement the InitializationMethod trait
/**
* Initialization method to initialize bias and weight.
* The init method will be called in Module.reset()
*/
trait InitializationMethod {
type Shape = Array[Int]
/**
* Initialize the given variable
*
* @param variable the variable to initialize
* @param dataFormat describe the meaning of each dimension of the variable
*/
def init[T](variable: Tensor[T], dataFormat: VariableFormat)
(implicit ev: TensorNumeric[T]): Unit
}
The RandomUniform code should give you a good sense of how to implement this trait.
_Python Custom initialization method in python is not supported right now.