Function rurtle::lex::tokenize
[−]
[src]
pub fn tokenize(input: &str) -> Result<VecDeque<MetaToken>, LexError>
Split the input String into single tokens. Strings in the input source are returned as a single token.