Theemo / @theemo/sync / LexerConfig

Interface: LexerConfig

Defined in: lexer/config.ts:7

The lexer config is used for you to configure the tokens to what they mean for you and to further process them

Properties

classifyToken()?

optional classifyToken: (token, tokens) => Token;

Defined in: lexer/config.ts:33

Describe your tokens:

• What's the type?
• What's the color scheme?

Parameters

Parameter	Type
token	Token
tokens	{ normalized: TokenCollection; raw: TokenCollection; }
tokens.normalized	TokenCollection
tokens.raw	TokenCollection

Returns

Token

---

filterToken()?

optional filterToken: (token, tokens) => boolean;

Defined in: lexer/config.ts:51

Filter callback to only keep the tokens you need.

Parameters

Parameter	Type
token	Token
tokens	{ classified: TokenCollection; normalized: TokenCollection; raw: TokenCollection; }
tokens.classified	TokenCollection
tokens.normalized	TokenCollection
tokens.raw	TokenCollection

Returns

boolean

Example

You may want to keep only purpose tokens, use this:

filterToken(token) {
  return token.type === 'purpose';
}

---

normalizeToken()?

optional normalizeToken: (token, tokens) => Token;

Defined in: lexer/config.ts:25

This is to normalize tokens and remove some glibberish off of it. Comes with a default, if you don't provide one (see in the example)

Parameters

Parameter	Type
token	Token
tokens	{ raw: TokenCollection; }
tokens.raw	TokenCollection

Returns

Token

Example

Here is how to remove any whitespace from token names:

normalizeToken(token: Token): Token {
  return {
    ...token,
    name: normalized.name.replace(/\s/g, '')
  };
}