Class JcsegTokenizer

  • All Implemented Interfaces:
    Closeable, AutoCloseable

    public class JcsegTokenizer
    extends org.apache.lucene.analysis.Tokenizer

    here is the documentation from Tokenizer A Tokenizer is a TokenStream whose input is a Reader.

    This is an abstract class; subclasses must override incrementToken()

    NOTE: Subclasses overriding incrementToken() must call AttributeSource.clearAttributes() before setting attributes

    lucene invoke Tokenizer#setReader(Reader input) to set the inputPending after invoke the reset, global object input will be available

    Jcseg tokenizer for lucene on or after 5.1.0

    Author:
    chenxin
    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

        org.apache.lucene.util.AttributeSource.State
    • Field Summary

      • Fields inherited from class org.apache.lucene.analysis.Tokenizer

        input
      • Fields inherited from class org.apache.lucene.analysis.TokenStream

        DEFAULT_TOKEN_ATTRIBUTE_FACTORY
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void end()  
      boolean incrementToken()  
      void reset()  
      • Methods inherited from class org.apache.lucene.analysis.Tokenizer

        close, correctOffset, setReader
      • Methods inherited from class org.apache.lucene.util.AttributeSource

        addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
    • Method Detail

      • incrementToken

        public final boolean incrementToken()
                                     throws IOException
        Specified by:
        incrementToken in class org.apache.lucene.analysis.TokenStream
        Throws:
        IOException
      • end

        public void end()
                 throws IOException
        Overrides:
        end in class org.apache.lucene.analysis.TokenStream
        Throws:
        IOException
      • reset

        public void reset()
                   throws IOException
        Overrides:
        reset in class org.apache.lucene.analysis.Tokenizer
        Throws:
        IOException