com.intel.analytics.bigdl.example.utils

TextClassifier

class TextClassifier extends Serializable

This example use a (pre-trained GloVe embedding) to convert word to vector, and uses it to train a text classification model on the 20 Newsgroup dataset with 20 different categories. This model can achieve around 90% accuracy after 2 epochs training.

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. TextClassifier
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TextClassifier(param: AbstractTextClassificationParams)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. def analyzeTexts(dataRdd: RDD[(String, Float)]): (Map[String, WordMeta], Map[Float, Array[Float]])

    Go through the whole data set to gather some meta info for the tokens.

    Go through the whole data set to gather some meta info for the tokens. Tokens would be discarded if the frequency ranking is less then maxWordsNum

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  8. def buildModel(classNum: Int): Sequential[Float]

    Return a text classification model with the specific num of class

  9. def buildWord2Vec(word2Meta: Map[String, WordMeta]): Map[Float, Array[Float]]

    Load the pre-trained word2Vec

    Load the pre-trained word2Vec

    returns

    A map from word to vector

  10. def buildWord2VecWithIndex(word2Meta: Map[String, Int]): Map[Float, Array[Float]]

    Load the pre-trained word2Vec

    Load the pre-trained word2Vec

    returns

    A map from word to vector

  11. var classNum: Int

  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  13. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  15. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  16. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  17. def getData(sc: SparkContext): (Array[RDD[(Array[Array[Float]], Float)]], Map[String, WordMeta], Map[Float, Array[Float]])

    Create train and val RDDs from input

  18. val gloveDir: String

  19. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  20. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  21. val log: Logger

  22. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  23. final def notify(): Unit

    Definition Classes
    AnyRef

  24. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  25. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  26. val textDataDir: String

  27. def toString(): String

    Definition Classes
    AnyRef → Any

  28. def train(): Unit

    Start to train the text classification model

  29. def trainFromData(sc: SparkContext, rdds: Array[RDD[(Array[Array[Float]], Float)]]): Module[Float]

    Train the text classification model with train and val RDDs

  30. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  31. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  32. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped