public class JcsegTokenizer
extends org.apache.lucene.analysis.Tokenizer
here is the documentation from Tokenizer
A Tokenizer is a TokenStream whose input is a Reader.
This is an abstract class; subclasses must override incrementToken()
NOTE: Subclasses overriding incrementToken()
must
call AttributeSource.clearAttributes()
before setting attributes
lucene invoke Tokenizer#setReader(Reader input) to set the inputPending after invoke the reset, global object input will be available
jcseg tokennizer for lucene on or after 5.1.0
Constructor and Description |
---|
JcsegTokenizer(ISegment.Type type,
SegmenterConfig config,
ADictionary dic) |
Modifier and Type | Method and Description |
---|---|
void |
end() |
boolean |
incrementToken() |
void |
reset() |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public JcsegTokenizer(ISegment.Type type, SegmenterConfig config, ADictionary dic) throws IOException
IOException
public final boolean incrementToken() throws IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
IOException
public void end() throws IOException
end
in class org.apache.lucene.analysis.TokenStream
IOException
public void reset() throws IOException
reset
in class org.apache.lucene.analysis.Tokenizer
IOException
Copyright © 2019. All rights reserved.