Package org.lionsoul.jcseg.analyzer
Class JcsegTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.lionsoul.jcseg.analyzer.JcsegTokenizer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public class JcsegTokenizer extends org.apache.lucene.analysis.Tokenizer
here is the documentation from
Tokenizer
A Tokenizer is a TokenStream whose input is a Reader.This is an abstract class; subclasses must override
incrementToken()
NOTE: Subclasses overriding
incrementToken()
must callAttributeSource.clearAttributes()
before setting attributeslucene invoke Tokenizer#setReader(Reader input) to set the inputPending after invoke the reset, global object input will be available
Jcseg tokenizer for lucene on or after 5.1.0
- Author:
- chenxin
-
-
Constructor Summary
Constructors Constructor Description JcsegTokenizer(ISegment.Type type, SegmenterConfig config, ADictionary dic)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
end()
boolean
incrementToken()
void
reset()
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Constructor Detail
-
JcsegTokenizer
public JcsegTokenizer(ISegment.Type type, SegmenterConfig config, ADictionary dic)
-
-
Method Detail
-
incrementToken
public final boolean incrementToken() throws IOException
- Specified by:
incrementToken
in classorg.apache.lucene.analysis.TokenStream
- Throws:
IOException
-
end
public void end() throws IOException
- Overrides:
end
in classorg.apache.lucene.analysis.TokenStream
- Throws:
IOException
-
reset
public void reset() throws IOException
- Overrides:
reset
in classorg.apache.lucene.analysis.Tokenizer
- Throws:
IOException
-
-