Lexeme

class understand.Lexeme

Bases: object

A token recieved from a lexer.

It has an associated range represented with line_begin, column_begin, line_end, and column_end. A lexeme also has the text for the range and a token describing the kind (comment, whitespace, etc). The inactive method can be used to test if the lexeme is in inactive code.

If a corresponding entity and/or reference exists for the lexeme, it can be retrieved with ent and ref respectively. Lexemes can be navigated with next and previous.

Lexeme line and column numbers are 1-based.

Methods Summary

column_begin	Return the beginning column number of the lexeme.
column_end	Return the ending column number of the lexeme.
ent	Return the entity associated with the lexeme or None if none.
inactive	Return True if the lexeme is part of inactive code.
line_begin	Return the beginning line number of the lexeme.
line_end	Return the ending line number of the lexeme.
next	Return the next lexeme, or None if no lexemes remain.
previous	Return the previous lexeme, or None if beginning of file.
ref	Return the reference associated with the lexeme, or None if none.
text	Return the text for the lexeme, which may be empty ("").
token	Return the token kind of the lexeme.

Methods Documentation

column_begin() → int

Return the beginning column number of the lexeme.

Return type:

int

Returns:

the beginning column

column_end() → int

Return the ending column number of the lexeme.

Return type:

int

Returns:

the ending column

ent() → understand.Ent

Return the entity associated with the lexeme or None if none.

Return type:

understand.Ent

Returns:

the associated entity or None if none

inactive() → bool

Return True if the lexeme is part of inactive code.

Return type:

bool

Returns:

True if the lexeme is part of inactive code

line_begin() → int

Return the beginning line number of the lexeme.

Return type:

int

Returns:

the beginning line

line_end() → int

Return the ending line number of the lexeme.

Return type:

int

Returns:

the ending line

next([ignore_whitespace[, ignore_comments]]) → understand.Lexeme

Return the next lexeme, or None if no lexemes remain.

Parameters:

• ignore_whitespace (bool or None) – if True, skip whitespace lexemes. False by default.

• ignore_comments (bool or None) – if True, skip comment lexemes. False by default.

Return type:

understand.Lexeme

Returns:

the next lexeme if it exists

previous([ignore_whitespace[, ignore_comments]]) → understand.Lexeme

Return the previous lexeme, or None if beginning of file.

Parameters:

• ignore_whitespace (bool or None) – if True, skip whitespace lexemes. False by default.

• ignore_comments (bool or None) – if True, skip comment lexemes. False by default.

Return type:

understand.Lexeme

Returns:

the previous lexeme if it exists

ref() → understand.Ref

Return the reference associated with the lexeme, or None if none.

Return type:

understand.Ref

Returns:

the associated reference or None if none

text() → str

Return the text for the lexeme, which may be empty (“”).

Return type:

str

Returns:

the associated text which may be empty

token() → str

Return the token kind of the lexeme.

Return type:

str

Returns:

a token value

Values include:

• Comment

• Continuation

• EndOfStatement

• Identifier

• Keyword

• Label

• Literal

• Newline

• Operator

• Preprocessor

• Punctuation

• String

• Whitespace

• Indent

• Dedent