Tokenisation¶

`ratus.token`¶

class ratus.token.Token(token_type: ratus.token.TokenType, lexeme: str)[source]¶: Representation of a token.

class ratus.token.TokenLiteral(token_type: ratus.token.TokenType, lexeme: str, literal: Any)[source]¶: Representation of a literal.

class ratus.token.TokenType[source]¶: Token type.

exception ratus.token.TokeniserError[source]¶: Error raised in the tokeniser.