public class JcsegTokenizer
extends org.apache.lucene.analysis.Tokenizer
here is the documentation from org.apache.lucene.analysis.tokenizer
A Tokenizer is a TokenStream whose input is a Reader.
This is an abstract class; subclasses must override incrementToken()
NOTE: Subclasses overriding incrementToken()
must
call AttributeSource#clearAttributes()
before setting attributes
lucene invoke Tokenizer#setReader(Reader input) to set the inputPending after invoke the reset, global object input will be available
jcseg tokennizer for lucene on or after 5.1.0
Constructor and Description |
---|
JcsegTokenizer(int mode,
JcsegTaskConfig config,
ADictionary dic) |
Modifier and Type | Method and Description |
---|---|
boolean |
incrementToken() |
void |
reset() |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
public JcsegTokenizer(int mode, JcsegTaskConfig config, ADictionary dic) throws JcsegException, IOException
JcsegException
IOException
public final boolean incrementToken() throws IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
IOException
public void reset() throws IOException
reset
in class org.apache.lucene.analysis.Tokenizer
IOException
Copyright © 2017. All Rights Reserved.