Sparse Layers

SparseLinear

Scala:

``````val module = SparseLinear(
inputSize,
outputSize,
withBias = true,
backwardStart: Int = -1,
backwardLength: Int = -1,
wRegularizer = null,
bRegularizer = null,
initWeight = null,
initBias = null,
``````

Python:

``````module = SparseLinear(
input_size,
output_size,
init_method="default",
with_bias=True,
backwardStart=-1,
backwardLength=-1,
wRegularizer=None,
bRegularizer=None,
init_weight=None,
init_bias=None,
``````

SparseLinear is the sparse version of module Linear. SparseLinear has two different from Linear: firstly, SparseLinear's input Tensor is a SparseTensor. Secondly, SparseLinear doesn't backward gradient to next layer in the backpropagation by default, as the gradInput of SparseLinear is useless and very big in most cases.

But, considering model like Wide&Deep, we provide backwardStart and backwardLength to backward part of the gradient to next layer.

Scala example:

``````import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor._

val module = SparseLinear(1000, 5)

val input = Tensor.sparse(Array(Array(0, 0, 0, 1, 1, 1), Array(1, 5, 300, 2, 100, 500)),
Array(1f, 3f, 5f, 2f, 4f, 6f),
Array(2, 1000))

println(module.forward(input))
``````

Gives the output,

``````0.047791008 0.069045454 0.020120896 0.019826084 0.10610865
-0.059406646    -0.13536823 -0.13861635 0.070304416 0.009570055
[com.intel.analytics.bigdl.tensor.DenseTensor\$mcF\$sp of size 2x5]
``````

Python example:

``````from bigdl.nn.layer import *
from bigdl.util.common import *
import numpy as np

module = SparseLinear(1000, 5)

input = JTensor.sparse(
np.array([1, 3, 5, 2, 4, 6]),
np.array([0, 0, 0, 1, 1, 1, 1, 5, 300, 2, 100, 500]),
np.array([2, 1000]))

print(module.forward(input))
``````

Gives the output,

``````[[ 10.09569263 -10.94844246  -4.1086688    1.02527523  11.80737209]
[  7.9651413    9.7131443  -10.22719955   0.02345783  -3.74368906]]
``````

SparseJoinTable

Scala:

``````val module = SparseJoinTable(dimension)
``````

Python:

``````module = SparseLinear(dimension)
``````

Experimental layer.

Sparse version of JoinTable. Backward just pass the origin gradOutput back to the next layers without split. So this layer may just works in Wide&Deep like models.

Scala example:

``````import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor._
import com.intel.analytics.bigdl.utils._

val module = SparseJoinTable(2)

val input1 = Tensor.sparse(Array(Array(0, 0, 0, 1, 1, 1), Array(1, 2, 3, 2, 3, 4)),
Array(1f, 2f, 3f, 4f, 5f, 6f),
Array(2, 5))
val input2 = Tensor.sparse(Array(Array(0, 0, 0, 1, 1, 1), Array(2, 3, 4, 1, 2, 3)),
Array(7f, 8f, 9f, 10f, 11f, 12f),
Array(2, 5))

println(module.forward(T(input1, input2)))
``````

Gives the output,

``````(0, 1) : 1.0
(0, 2) : 2.0
(0, 3) : 3.0
(0, 7) : 7.0
(0, 8) : 8.0
(0, 9) : 9.0
(1, 2) : 4.0
(1, 3) : 5.0
(1, 4) : 6.0
(1, 6) : 10.0
(1, 7) : 11.0
(1, 8) : 12.0
[com.intel.analytics.bigdl.tensor.SparseTensor of size 2x10]
``````

Python example:

``````from bigdl.nn.layer import *
from bigdl.util.common import *
import numpy as np

module = SparseJoinTable(2)

input1 = JTensor.sparse(np.array([1, 2, 3, 4, 5, 6]),
np.array([0, 0, 0, 1, 1, 1, 1, 2, 3, 2, 3, 4]),
np.array([2, 5]))
input2 = JTensor.sparse(np.array([7, 8, 9, 10, 11, 12]),
np.array([0, 0, 0, 1, 1, 1, 2, 3, 4, 1, 2, 3]),
np.array([2, 5]))

print(module.forward([input1, input2]))
``````

Gives the output, this output is a dense numpy array, due to we couldn't pick SparseTensor back to python currently.

``````[[  0.   1.   2.   3.   0.   0.   0.   7.   8.   9.]
[  0.   0.   4.   5.   6.   0.  10.  11.  12.   0.]]
``````

DenseToSparse

Scala:

``````val module = DenseToSparse()
``````

Python:

``````module = DenseToSparse()
``````

Convert DenseTensor to SparseTensor.

Scala example:

``````import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric.NumericFloat
import com.intel.analytics.bigdl.nn._
import com.intel.analytics.bigdl.tensor._

val module = DenseToSparse()

val input = Tensor(2, 3)
input.setValue(1, 1, 1)
input.setValue(2, 2, 2)

println(module.forward(input))
``````

Gives the output,

``````(0, 0) : 1.0
(1, 1) : 2.0
[com.intel.analytics.bigdl.tensor.SparseTensor of size 2x3]
``````

Python example:

``````from bigdl.nn.layer import *
from bigdl.util.common import *
import numpy as np

module = DenseToSparse()

input = np.zeros([2, 3])
input[0, 0] = 1
input[1, 1] = 2

print(module.forward(input))
``````

Gives the output, this output is a dense numpy array, due to we couldn't pick SparseTensor back to python currently.

``````[[ 1.  0.  0.]
[ 0.  2.  0.]]
``````