Definition of the Lexer machine to process raw DC file string data into a stream of lexical tokens for the DC parser.