Lexer

This step runs a lexical analysis based on your input to make tokens understandable to you. The lexer will receive a TokenCollection from the reader modifies the set of tokens and returns a TokenCollection that will be the input to the writer.

For the modification, the lexer applies these steps, that act upon your configuration.

The output of one step is the input to the next step. If you classify a token, you can use these information to filter them afterwards.

See your configuration options in detail with samples:

• normalizeToken()
• classifyToken()
• filterToken()