AbstractDataSet

A set of data which is used in the model optimization process. The dataset can be access in a random data sample sequence. In the training process, the data sequence is a looped endless sequence. While in the validation process, the data sequence is a limited length sequence. User can use the data() method to get the data sequence.

The sequence of the data is not fixed. It can be changed by the shuffle() method.

User can create a dataset from a RDD, an array and a folder, etc. The DataSet object provides many factory methods.

D: Data type
DataSequence: Represent a sequence of data

Linear Supertypes

AnyRef, Any

Known Subclasses

CachedDistriDataSet, DistributedDataSet, LocalArrayDataSet, LocalDataSet

Abstract Value Members

abstract def data(train: Boolean): DataSequence

Get a sequence of data
Get a sequence of data
train
if the data is used in train. If yes, the data sequence is a looped endless sequence, or it has a limited length.
returns
data sequence
abstract def shuffle(): Unit

Change the order of the data sequence from the data set
abstract def size(): Long

Total size of the data set
abstract def transform[C](transformer: Transformer[D, C])(implicit arg0: ClassTag[C]): DataSet[C]

Helper function to transform the data type in the data set.

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
def ->[C](transformer: Transformer[D, C])(implicit arg0: ClassTag[C]): DataSet[C]

Helper function to transform the data type in the data set.
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toDistributed(): DistributedDataSet[D]

Convert current DataSet to a distributed DataSet, in which we use a RDD to represent the data sequence.
def toLocal(): LocalDataSet[D]

Convert current DataSet to a local DataSet, in which we use an iterator to represent the data sequence.
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package dataset

trait AbstractDataSet[D, DataSequence] extends AnyRef

Abstract Value Members

abstract def data(train: Boolean): DataSequence

abstract def shuffle(): Unit

abstract def size(): Long

abstract def transform[C](transformer: Transformer[D, C])(implicit arg0: ClassTag[C]): DataSet[C]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

def ->[C](transformer: Transformer[D, C])(implicit arg0: ClassTag[C]): DataSet[C]

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toDistributed(): DistributedDataSet[D]

def toLocal(): LocalDataSet[D]

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped