Lexer

Lexical tokenization is conversion of a text into meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping symbols, data types and language keywords. Lexical tokenization is related to the type of tokenization used in large language models (LLMs) but with two differences. First, lexical tokenization is usually based on a lexical grammar, whereas LLM tokenizers are usually probability-based. Second, LLM tokenizers perform a second step that converts the tokens into numerical values.

Spotify · Apple Music · SoundCloud

FAQ

Where can I find Lexer live shows?

Open Lexer in the SoundOff app for personalized upcoming show picks based on your own concert history.

How do SoundOff fans rank Lexer?

SoundOff uses head-to-head comparisons instead of star ratings. Fans who logged Lexer compare sets to build a personal ranking over time.