org.apache.spark.ml

DLEstimator

class DLEstimator[T] extends com.intel.analytics.bigdl.dlframes.DLEstimator[T]

Deprecated. Please refer to package com.intel.analytics.bigdl.dlframes.

DLEstimator helps to train a BigDL Model with the Spark ML Estimator/Transfomer pattern, thus Spark users can conveniently fit BigDL into Spark ML pipeline.

DLEstimator supports feature and label data in the format of Array[Double], Array[Float], org.apache.spark.mllib.linalg.{Vector, VectorUDT}, org.apache.spark.ml.linalg.{Vector, VectorUDT}, Double and Float.

User should specify the feature data dimensions and label data dimensions via the constructor parameters featureSize and labelSize respectively. Internally the feature and label data are converted to BigDL tensors, to further train a BigDL model efficiently.

For details usage, please refer to examples in package com.intel.analytics.bigdl.example.MLPipeline

Annotations
@deprecated
Deprecated

(Since version 0.5.0)

Linear Supertypes
com.intel.analytics.bigdl.dlframes.DLEstimator[T], DLParams[T], HasBatchSize, VectorCompatibility, HasPredictionCol, HasPredictionCol, HasFeaturesCol, HasFeaturesCol, DLEstimatorBase[com.intel.analytics.bigdl.dlframes.DLEstimator[T], com.intel.analytics.bigdl.dlframes.DLModel[T]], HasLabelCol, Estimator[com.intel.analytics.bigdl.dlframes.DLModel[T]], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. DLEstimator
  2. DLEstimator
  3. DLParams
  4. HasBatchSize
  5. VectorCompatibility
  6. HasPredictionCol
  7. HasPredictionCol
  8. HasFeaturesCol
  9. HasFeaturesCol
  10. DLEstimatorBase
  11. HasLabelCol
  12. Estimator
  13. PipelineStage
  14. Logging
  15. Params
  16. Serializable
  17. Serializable
  18. Identifiable
  19. AnyRef
  20. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DLEstimator(model: Module[T], criterion: Criterion[T], featureSize: Array[Int], labelSize: Array[Int], uid: String = "DLEstimator")(implicit arg0: ClassTag[T], ev: TensorNumeric[T])

    model

    BigDL module to be optimized

    criterion

    BigDL criterion method

    featureSize

    The size (Tensor dimensions) of the feature data. e.g. an image may be with width * height = 28 * 28, featureSize = Array(28, 28).

    labelSize

    The size (Tensor dimensions) of the label data.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def $[T](param: Param[T]): T

    Attributes
    protected
    Definition Classes
    Params
  5. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  6. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. final val batchSize: Param[Int]

    Definition Classes
    HasBatchSize
  9. final def clear(param: Param[_]): DLEstimator.this.type

    Attributes
    protected
    Definition Classes
    Params
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def copy(extra: ParamMap): com.intel.analytics.bigdl.dlframes.DLEstimator[T]

    Definition Classes
    DLEstimator → DLEstimatorBase → Estimator → PipelineStage → Params
  12. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Attributes
    protected
    Definition Classes
    Params
  13. val criterion: Criterion[T]

    BigDL criterion method

    BigDL criterion method

    Definition Classes
    DLEstimatorDLEstimator
  14. final def defaultCopy[T <: Params](extra: ParamMap): T

    Attributes
    protected
    Definition Classes
    Params
  15. final val endWhen: Param[Trigger]

    When to stop the training, passed in a Trigger.

    When to stop the training, passed in a Trigger. E.g. Trigger.maxIterations

    Definition Classes
    DLParams
  16. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  17. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  18. def explainParam(param: Param[_]): String

    Definition Classes
    Params
  19. def explainParams(): String

    Definition Classes
    Params
  20. final def extractParamMap(): ParamMap

    Definition Classes
    Params
  21. final def extractParamMap(extra: ParamMap): ParamMap

    Definition Classes
    Params
  22. final val featuresCol: Param[String]

    Definition Classes
    HasFeaturesCol
  23. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  24. def fit(dataFrame: DataFrame): com.intel.analytics.bigdl.dlframes.DLModel[T]

    Definition Classes
    DLEstimatorBase → Estimator
  25. def fit(dataset: DataFrame, paramMaps: Array[ParamMap]): Seq[com.intel.analytics.bigdl.dlframes.DLModel[T]]

    Definition Classes
    Estimator
  26. def fit(dataset: DataFrame, paramMap: ParamMap): com.intel.analytics.bigdl.dlframes.DLModel[T]

    Definition Classes
    Estimator
  27. def fit(dataset: DataFrame, firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): com.intel.analytics.bigdl.dlframes.DLModel[T]

    Definition Classes
    Estimator
    Annotations
    @varargs()
  28. final def get[T](param: Param[T]): Option[T]

    Definition Classes
    Params
  29. def getBatchSize: Int

    Definition Classes
    HasBatchSize
  30. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  31. def getConvertFunc(colType: DataType): (Row, Int) ⇒ Seq[AnyVal]

    Get conversion function to extract data from original DataFrame Default: 0

    Get conversion function to extract data from original DataFrame Default: 0

    Attributes
    protected
    Definition Classes
    DLParams
  32. final def getDefault[T](param: Param[T]): Option[T]

    Definition Classes
    Params
  33. def getEndWhen: Trigger

    Definition Classes
    DLParams
  34. final def getFeaturesCol: String

    Definition Classes
    HasFeaturesCol
  35. final def getLabelCol: String

    Definition Classes
    HasLabelCol
  36. def getLearningRate: Double

    Definition Classes
    DLParams
  37. def getLearningRateDecay: Double

    Definition Classes
    DLParams
  38. def getMaxEpoch: Int

    Definition Classes
    DLParams
  39. def getOptimMethod: OptimMethod[T]

    Definition Classes
    DLParams
  40. final def getOrDefault[T](param: Param[T]): T

    Definition Classes
    Params
  41. def getParam(paramName: String): Param[Any]

    Definition Classes
    Params
  42. final def getPredictionCol: String

    Definition Classes
    HasPredictionCol
  43. def getTrainSummary: Option[TrainSummary]

    Definition Classes
    DLEstimator
  44. def getValidationSummary: Option[ValidationSummary]

    Statistics (LearningRate, Loss, Throughput, Parameters) collected during training for the validation data if validation data is set, which can be used for visualization via Tensorboard.

    Statistics (LearningRate, Loss, Throughput, Parameters) collected during training for the validation data if validation data is set, which can be used for visualization via Tensorboard. Use setValidationSummary to enable validation logger. Then the log will be saved to logDir/appName/ as specified by the parameters of validationSummary.

    Default: None

    Definition Classes
    DLEstimator
  45. def getVectorSeq(row: Row, colType: DataType, index: Int): Seq[AnyVal]

    Definition Classes
    VectorCompatibility
  46. final def hasDefault[T](param: Param[T]): Boolean

    Definition Classes
    Params
  47. def hasParam(paramName: String): Boolean

    Definition Classes
    Params
  48. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  49. def internalFit(dataFrame: DataFrame): com.intel.analytics.bigdl.dlframes.DLModel[T]

    Attributes
    protected
    Definition Classes
    DLEstimator → DLEstimatorBase
  50. final def isDefined(param: Param[_]): Boolean

    Definition Classes
    Params
  51. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  52. final def isSet(param: Param[_]): Boolean

    Definition Classes
    Params
  53. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  54. final val labelCol: Param[String]

    Definition Classes
    HasLabelCol
  55. val labelSize: Array[Int]

    The size (Tensor dimensions) of the label data.

    The size (Tensor dimensions) of the label data.

    Definition Classes
    DLEstimatorDLEstimator
  56. final val learningRate: DoubleParam

    learning rate for the optimizer in the DLEstimator.

    learning rate for the optimizer in the DLEstimator. Default: 0.001

    Definition Classes
    DLParams
  57. final val learningRateDecay: DoubleParam

    learning rate decay for each iteration.

    learning rate decay for each iteration. Default: 0

    Definition Classes
    DLParams
  58. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  59. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  60. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  61. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  62. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  63. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  64. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  65. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  67. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  69. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  70. final val maxEpoch: IntParam

    Number of max Epoch for the training, an epoch refers to a traverse over the training data Default: 50

    Number of max Epoch for the training, an epoch refers to a traverse over the training data Default: 50

    Definition Classes
    DLParams
  71. val model: Module[T]

    BigDL module to be optimized

    BigDL module to be optimized

    Definition Classes
    DLEstimatorDLEstimator
  72. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  73. final def notify(): Unit

    Definition Classes
    AnyRef
  74. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  75. final val optimMethod: Param[OptimMethod[T]]

    optimization method to be used.

    optimization method to be used. BigDL supports many optimization methods like Adam, SGD and LBFGS. Refer to package com.intel.analytics.bigdl.optim for all the options. Default: SGD

    Definition Classes
    DLParams
  76. lazy val params: Array[Param[_]]

    Definition Classes
    Params
  77. final val predictionCol: Param[String]

    Definition Classes
    HasPredictionCol
  78. final def set(paramPair: ParamPair[_]): DLEstimator.this.type

    Attributes
    protected
    Definition Classes
    Params
  79. final def set(param: String, value: Any): DLEstimator.this.type

    Attributes
    protected
    Definition Classes
    Params
  80. final def set[T](param: Param[T], value: T): DLEstimator.this.type

    Attributes
    protected
    Definition Classes
    Params
  81. def setBatchSize(value: Int): DLEstimator.this.type

    Definition Classes
    DLEstimator
  82. final def setDefault(paramPairs: ParamPair[_]*): DLEstimator.this.type

    Attributes
    protected
    Definition Classes
    Params
  83. final def setDefault[T](param: Param[T], value: T): DLEstimator.this.type

    Attributes
    protected
    Definition Classes
    Params
  84. def setEndWhen(trigger: Trigger): DLEstimator.this.type

    Definition Classes
    DLEstimator
  85. def setFeaturesCol(featuresColName: String): DLEstimator.this.type

    Definition Classes
    DLEstimator
  86. def setLabelCol(labelColName: String): DLEstimator.this.type

    Definition Classes
    DLEstimator
  87. def setLearningRate(value: Double): DLEstimator.this.type

    Definition Classes
    DLEstimator
  88. def setLearningRateDecay(value: Double): DLEstimator.this.type

    Definition Classes
    DLEstimator
  89. def setMaxEpoch(value: Int): DLEstimator.this.type

    Definition Classes
    DLEstimator
  90. def setOptimMethod(value: OptimMethod[T]): DLEstimator.this.type

    Definition Classes
    DLEstimator
  91. def setPredictionCol(value: String): DLEstimator.this.type

    Definition Classes
    DLEstimator
  92. def setTrainSummary(value: TrainSummary): DLEstimator.this.type

    Statistics (LearningRate, Loss, Throughput, Parameters) collected during training for the training data, which can be used for visualization via Tensorboard.

    Statistics (LearningRate, Loss, Throughput, Parameters) collected during training for the training data, which can be used for visualization via Tensorboard. Use setTrainSummary to enable train logger. Then the log will be saved to logDir/appName/train as specified by the parameters of TrainSummary.

    Default: Not enabled

    Definition Classes
    DLEstimator
  93. def setValidation(trigger: Trigger, validationDF: DataFrame, vMethods: Array[ValidationMethod[T]], batchSize: Int): DLEstimator.this.type

    Set a validate evaluation during training

    Set a validate evaluation during training

    trigger

    how often to evaluation validation set

    validationDF

    validate data set

    vMethods

    a set of validation method ValidationMethod

    batchSize

    batch size for validation

    returns

    this optimizer

    Definition Classes
    DLEstimator
  94. def setValidationSummary(value: ValidationSummary): DLEstimator.this.type

    Enable validation Summary

    Enable validation Summary

    Definition Classes
    DLEstimator
  95. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  96. def toString(): String

    Definition Classes
    Identifiable → AnyRef → Any
  97. def transformSchema(schema: StructType): StructType

    Definition Classes
    DLEstimator → PipelineStage
  98. def transformSchema(schema: StructType, logging: Boolean): StructType

    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  99. val uid: String

    Definition Classes
    DLEstimatorDLEstimator → Identifiable
  100. val validVectorTypes: Seq[VectorUDT]

    Definition Classes
    VectorCompatibility
  101. def validateDataType(schema: StructType, colName: String): Unit

    Validate if feature and label columns are of supported data types.

    Validate if feature and label columns are of supported data types. Default: 0

    Attributes
    protected
    Definition Classes
    DLParams
  102. def validateParams(schema: StructType): Unit

    Attributes
    protected
    Definition Classes
    DLEstimator
  103. def validateParams(): Unit

    Definition Classes
    Params
  104. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  105. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  106. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  107. def wrapBigDLModel(m: Module[T], featureSize: Array[Int]): DLModel[T]

    sub classes can extend the method and return required model for different transform tasks

    sub classes can extend the method and return required model for different transform tasks

    Attributes
    protected
    Definition Classes
    DLEstimatorDLEstimator

Inherited from DLParams[T]

Inherited from HasBatchSize

Inherited from VectorCompatibility

Inherited from HasPredictionCol

Inherited from HasPredictionCol

Inherited from HasFeaturesCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Estimator[com.intel.analytics.bigdl.dlframes.DLModel[T]]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Ungrouped