Package com.pnfsoftware.jeb.util.encoding.jflex

Class DefaultJFlexLexer

java.lang.Object

com.pnfsoftware.jeb.util.encoding.jflex.DefaultJFlexLexer

Direct Known Subclasses:

JavaLexer, PythonLexer

public abstract class DefaultJFlexLexer
extends Object

This is a default, and abstract implementation of a Lexer using JFLex with some utility methods that Lexers can implement. See flex examples at https://github.com/nordfalk/jsyntaxpane.

Field Summary

Fields

Modifier and Type	Field	Description
protected int	offset
protected int	tokenLength
protected long	tokenStart

Constructor Summary

Constructors

Constructor	Description
DefaultJFlexLexer()

Method Summary

Modifier and Type	Method	Description
void	parse(Reader in, List<Token> tokens)
List<Token>	parse(String s)
protected Token	token(TokenType type)	Create and return a Token of given type.
protected Token	token(TokenType type, int pairValue)	Create and return a Token of given type and pairValue.
protected Token	token(TokenType type, int tStart, int tLength, int newStart, int newLength)	Helper method to create and return a new Token from of TokenType tokenStart and tokenLength will be modified to the newStart and newLength params
protected Token	token(TokenType type, long start, int length)	Create and return a Token of given type from start with length offset is added to start
abstract long	yychar()	Return the char number from beginning of input stream.
abstract char	yycharat(int pos)	Returns the character at position pos from the matched text.
abstract int	yylength()	Returns the length of the matched text region.
abstract Token	yylex()	This is called to return the next Token from the Input Reader
abstract void	yyreset(Reader reader)	This will be called to reset the the lexer.
abstract String	yytext()	Returns the text matched by the current regular expression.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

tokenStart

protected long tokenStart

tokenLength

protected int tokenLength

offset

protected int offset

Constructor Details

DefaultJFlexLexer

public DefaultJFlexLexer()

Method Details

token

protected Token token(TokenType type,
 int tStart,
 int tLength,
 int newStart,
 int newLength)

Helper method to create and return a new Token from of TokenType tokenStart and tokenLength will be modified to the newStart and newLength params

token

protected Token token(TokenType type,
 long start,
 int length)

Create and return a Token of given type from start with length offset is added to start

token

protected Token token(TokenType type)

Create and return a Token of given type. start is obtained from yychar() and length from yylength() offset is added to start

token

protected Token token(TokenType type,
 int pairValue)

Create and return a Token of given type and pairValue. start is obtained from yychar() and length from yylength() offset is added to start

parse

public List<Token> parse(String s)
                  throws IOException

Throws:

IOException

parse

public void parse(Reader in,
 List<Token> tokens)
           throws IOException

Throws:

IOException

yyreset

public abstract void yyreset(Reader reader)

This will be called to reset the the lexer. This is created automatically by JFlex.

yylex

public abstract Token yylex()
                     throws IOException

This is called to return the next Token from the Input Reader

Returns:

next token, or null if no more tokens.

Throws:

IOException

yycharat

public abstract char yycharat(int pos)

Returns the character at position pos from the matched text. It is equivalent to yytext().charAt(pos), but faster

Parameters:

pos - the position of the character to fetch. A value from 0 to yylength()-1.

Returns:

the character at position pos

yylength

public abstract int yylength()

Returns the length of the matched text region. This method is automatically implemented by JFlex lexers

yytext

public abstract String yytext()

Returns the text matched by the current regular expression. This method is automatically implemented by JFlex lexers

yychar

public abstract long yychar()

Return the char number from beginning of input stream. This is NOT implemented by JFlex, so the code must be added to create this and return the private yychar field