com.intel.analytics.bigdl.parameters

AllReduceParameter

class AllReduceParameter[T] extends Serializable

Represent parameters stored on the block manager. In distributed optimization, we put parameters on block manager of spark. Each worker syncs parameters through the block manager. Block manager here serves as a parameter server.

A Tensor is sliced into partitionNum chunks and each chunk is assigned to a particular node (Spark executor). Likewise, gradients for each chunk are also assigned and stored on separate nodes. In this way, gradient aggregation and parameter updates can be performed independently for each chunk on separate nodes.

T

Tensor element type

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. AllReduceParameter
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AllReduceParameter(id: Long, partitionNum: Int, size: Int, paramOffset: Int = 1)(implicit arg0: ClassTag[T], ev: TensorNumeric[T])

    id

    distinguish from other parameters

    partitionNum

    how many partitions will use this parameter

    size

    size of the parameter (1D vector)

    paramOffset

    start index in the origin parameter.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def aggregateGradientPartition(avgNumbers: Int): Unit

    Retrieve gradients for the slice of the model that this node is responsible for from all the other nodes.

    Retrieve gradients for the slice of the model that this node is responsible for from all the other nodes. A new thread is created for each separate node. The gradients are then summed and then stored in decompressed form in gradientPartition.

    avgNumbers

    average numbers.

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  13. def getWeights(localParameter: Tensor[T]): FutureResult[Int]

    Use a fixed thread pool to launch a thread for each partition of the weights.

    Use a fixed thread pool to launch a thread for each partition of the weights. Each thread requests a partition of the weights from the Spark block manager and copies it into localParameter.

    localParameter

    The Tensor that will hold the retrieved weights.

    returns

    A FutureResult which contains a Future for each thread.

  14. lazy val gradientPartition: Tensor[T]

    Tensor to hold a slice of the global gradients.

  15. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  16. def init(parameter: Tensor[T])(implicit ev: TensorNumeric[T]): (Int, Int, Int)

    This method should be called on each RDD partition before parameter synchronization begins.

    This method should be called on each RDD partition before parameter synchronization begins. An empty gradient tensor is placed in the block manager that can be used to store gradients. A 1 / numPartition fraction of the parameter tensor is copied to the block manager as a compressed tensor.

    parameter

    A tensor representing the initial underlying weights of this AllReduceParameter

  17. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  18. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  19. final def notify(): Unit

    Definition Classes
    AnyRef
  20. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  21. val paramOffset: Int

    start index in the origin parameter.

  22. def putGradients(parameter: Tensor[T]): Unit

    Slice gradients learned from this partition of data into chunks, and mark each chunk to be sent to the appropriate parameter node, and put it in the block manager.

    Slice gradients learned from this partition of data into chunks, and mark each chunk to be sent to the appropriate parameter node, and put it in the block manager.

    parameter

    A Tensor that contains gradients computed on the entire model on a single partition of data.

  23. def sendWeightPartition(): Unit

    Put the portion of the weights that this partition is responsible for to the block manager.

    Put the portion of the weights that this partition is responsible for to the block manager. Weights are placed locally, then pulled when needed by other partitions.

  24. val size: Int

    size of the parameter (1D vector)

  25. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  26. def toString(): String

    Definition Classes
    AnyRef → Any
  27. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. lazy val weightPartition: Tensor[T]

    Tensor to hold a slice of the global weights.

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped