Class

com.intel.analytics.bigdl.example.utils

TextClassifier

Related Doc: package utils

Permalink

class TextClassifier extends Serializable

This example use a (pre-trained GloVe embedding) to convert word to vector, and uses it to train a text classification model on the 20 Newsgroup dataset with 20 different categories. This model can achieve around 90% accuracy after 2 epochs training.

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TextClassifier
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new TextClassifier(param: AbstractTextClassificationParams)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. def analyzeTexts(dataRdd: RDD[(String, Float)]): (Map[String, WordMeta], Map[Float, Array[Float]])

    Permalink

    Go through the whole data set to gather some meta info for the tokens.

    Go through the whole data set to gather some meta info for the tokens. Tokens would be discarded if the frequency ranking is less then maxWordsNum

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. def buildModel(classNum: Int): Sequential[Float]

    Permalink

    Return a text classification model with the specific num of class

  7. def buildWord2Vec(word2Meta: Map[String, WordMeta]): Map[Float, Array[Float]]

    Permalink

    Load the pre-trained word2Vec

    Load the pre-trained word2Vec

    returns

    A map from word to vector

  8. def buildWord2VecWithIndex(word2Meta: Map[String, Int]): Map[Float, Array[Float]]

    Permalink

    Load the pre-trained word2Vec

    Load the pre-trained word2Vec

    returns

    A map from word to vector

  9. var classNum: Int

    Permalink

  10. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  11. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  12. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  13. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  14. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  15. def getData(sc: SparkContext): (Array[RDD[(Array[Array[Float]], Float)]], Map[String, WordMeta], Map[Float, Array[Float]])

    Permalink

    Create train and val RDDs from input

  16. val gloveDir: String

    Permalink

  17. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  18. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  19. val log: Logger

    Permalink

  20. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  21. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  22. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  23. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  24. val textDataDir: String

    Permalink

  25. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  26. def train(): Unit

    Permalink

    Start to train the text classification model

  27. def trainFromData(sc: SparkContext, rdds: Array[RDD[(Array[Array[Float]], Float)]]): Module[Float]

    Permalink

    Train the text classification model with train and val RDDs

  28. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  29. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  30. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped