Module rurtle::lex

This module contains the lexical analyser for the Turtle language. It can split a Rurtle source string into single tokens, which can be later used by the interpreter.

Valid identifiers start with any (unicode) alphabetic character and may consist of any alpha-numeric character thereafter.

Strings have to be enclosed in double quotes ("), there are no strings in enclosed in lists. For example, this is valid: "Hello", this is not: [Hello]

Lists are enclosed in []-brackets.

Variables are prefixed by a colon (:) and otherwise follow the same rules as identifiers.

Structs

MetaToken

This struct contains a token and some additional meta information

Enums

LexError	Errors that may arise when lexing the input. The first member is always the line number.
Token	A Token represents a "atom" block of the input source.

Functions

tokenize

Split the input String into single tokens. Strings in the input source are returned as a single token.