Python API tutorial 4: lexers and lexemes¶

Most scripts work with the entities and references stored in the database, but sometimes you need the raw text of a file token by token. Understand gives you that through a file's lexer and the lexeme objects it yields.

Lexeme — a chunk of text that means something to the parser: a keyword, an identifier, a string, a comment, punctuation, and so on.
Lexer (understand.Lexer) — a stream of lexemes for a file.

Walking the stream, you can ask each lexeme for its text, its token type (Comment, Punctuation, Preprocessor, Identifier, …), the entity or reference it's associated with, and the line it's on.

Opening a lexer¶

Get a lexer from a file entity with file.lexer(...). The parameters, in order, are:

Parameter	Default	Meaning
`lookup_ents`	`True`	Associate lexemes with entities/references
`tabstop`	`1`	Deprecated — accepted for backward compatibility but ignored; tabs always count as 1 column
`show_inactive`	`True`	Include code disabled by the preprocessor
`expand_macros`	`False`	Expand macros in the token stream

Example: clean text of a file¶

Return a file's text with comments removed, inactive (preprocessed-out) code removed, and macros expanded:

import understand

db = understand.open("C:/sample project/sample_project.und")

def fileCleanText(file):
    returnString = ""
    # Open the file lexer with macros expanded and inactive code removed
    for lexeme in file.lexer(False, 1, False, True):
        if lexeme.token() != "Comment":
            # Append the text of every non-comment lexeme
            returnString += lexeme.text()
    return returnString

# Find the first file whose name contains 'test', then print it and its cleaned text
file = db.lookup(".test.", "file")[0]
print(file.longname())
print(fileCleanText(file))

How it works:

file.lexer(False, 1, False, True) opens the lexer with lookup_ents=False, tabstop=1 (ignored — kept only for signature compatibility), show_inactive=False (drop inactive code), and expand_macros=True (expand macros).
Iterating the lexer yields each Lexeme. lexeme.token() returns the token type as a string, and lexeme.text() returns the raw text.
db.lookup(".test.", "file") looks up entities by name — the name argument is a regular expression, so .test. matches any file whose name contains test. It returns a list; [0] takes the first match.

lookup() returns a list

db.lookup(...) can return an empty list. Guard the [0] (or check len(...)) before indexing if the name might not match anything.

Next¶

You can now read source at the token level. Next you'll turn a script into a plugin that draws a graph.

← Tutorial 3: entities, references, and filters · → Tutorial 5: graphs