Tokenizer is one of Understand's built-in reports — run it on your own code from the GUI, or headlessly with und report.
Tokenizer
Languages: AnyTargets: Classes, Files, Functions
Display the tokens generated for a file or entity.
This report shows each token as generated by the lexer, its token kind and line number. Select a file or an entity to see its tokens.
See also the Tokenizer Table ireport which displays the same information with a table.
