com.intel.analytics.bigdl.dataset

AbstractDataSet

trait AbstractDataSet[D, DataSequence] extends AnyRef

A set of data which is used in the model optimization process. The dataset can be access in a random data sample sequence. In the training process, the data sequence is a looped endless sequence. While in the validation process, the data sequence is a limited length sequence. User can use the data() method to get the data sequence.

The sequence of the data is not fixed. It can be changed by the shuffle() method.

User can create a dataset from a RDD, an array and a folder, etc. The DataSet object provides many factory methods.

D

Data type

DataSequence

Represent a sequence of data

Linear Supertypes
AnyRef, Any
Known Subclasses
CachedDistriDataSet, DistributedDataSet, LocalArrayDataSet, LocalDataSet
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. AbstractDataSet
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def data(train: Boolean): DataSequence

    Get a sequence of data

    Get a sequence of data

    train

    if the data is used in train. If yes, the data sequence is a looped endless sequence, or it has a limited length.

    returns

    data sequence

  2. abstract def shuffle(): Unit

    Change the order of the data sequence from the data set

  3. abstract def size(): Long

    Total size of the data set

    Total size of the data set

    returns

  4. abstract def transform[C](transformer: Transformer[D, C])(implicit arg0: ClassTag[C]): DataSet[C]

    Helper function to transform the data type in the data set.

    Helper function to transform the data type in the data set.

    C
    transformer
    returns

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. def ->[C](transformer: Transformer[D, C])(implicit arg0: ClassTag[C]): DataSet[C]

    Helper function to transform the data type in the data set.

    Helper function to transform the data type in the data set.

    C
    transformer
    returns

  5. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  6. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  13. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  14. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  15. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  16. final def notify(): Unit

    Definition Classes
    AnyRef

  17. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  18. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  19. def toDistributed(): DistributedDataSet[D]

    Convert current DataSet to a distributed DataSet, in which we use a RDD to represent the data sequence.

    Convert current DataSet to a distributed DataSet, in which we use a RDD to represent the data sequence.

    returns

  20. def toLocal(): LocalDataSet[D]

    Convert current DataSet to a local DataSet, in which we use an iterator to represent the data sequence.

    Convert current DataSet to a local DataSet, in which we use an iterator to represent the data sequence.

    returns

  21. def toString(): String

    Definition Classes
    AnyRef → Any

  22. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  23. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  24. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped