Racket - define one character with token-char - parsing

Question

I am working on a project for a class and we are tasked with writing a scanner for numbers, symbols, comments, arithmetic operators, parenthesis, and EOF in both Python and Racket. I am working on the racket version and I have written the following line to define one or more character as a symbol:
[(any-char) (token-CHAR (string->character lexeme))]
I have the following line to define on or more digits as a number:
[(:+ digit) (token-NUM (string->number lexeme))]
I am very new to Racket, this is my third program, so I am not exactly sure how to approach this, so any suggestions are greatly appreciated. I have scoured the Racket documentation, but I wasn't able to find what I was looking for.
Thanks!

Answer 1

Here is a minimal getting-started example - heavily commented.
#lang racket
;;; IMPORT
;; Import the lexer tools
(require parser-tools/yacc
parser-tools/lex
(prefix-in : parser-tools/lex-sre) ; names from lex-sre are prefixed with :
; to avoid name collisions
syntax/readerr)
;;; REGULAR EXPRESSIONS
;; Names for regular expressions matching letters and digits.
;; Note that :or are prefixed with a : due to (prefix-in : ...) above
(define-lex-abbrevs
[letter (:or (:/ "a" "z") (:/ #\A #\Z) )]
[digit (:/ #\0 #\9)])
;;; TOKENS
;; Tokens such as numbers (and identifiers and strings) carry a value
;; In the example only the NUMBER token is used, but you may need more.
(define-tokens value-tokens (NUMBER IDENTIFIER STRING))
;; Tokens that don't carry a value.
(define-empty-tokens op-tokens (newline := = < > + - * / ^ EOF))
;;; LEXER
;; Here the lexer (aka the scanner) is defined.
;; The construct lexer-src-pos evaluates to a function which scans an input port
;; returning one position-token at a time.
;; A position token contains besides the actual token also source location information
;; (i.e. you can see where in the file the token was read)
(define lex
(lexer-src-pos
[(eof) ; input: eof of file
'EOF] ; output: the symbol EOF
[(:or #\tab #\space #\newline) ; input: whitespace
(return-without-pos (lex input-port))] ; output: the next token
; (i.e. skip the whitespace)
[#\newline ; input: newline
(token-newline)] ; ouput: a newline-token
; ; note: (token-newline) returns 'newline
[(:or ":=" "+" "-" "*" "/" "^" "<" ">" "=") ; input: an operator
(string->symbol lexeme)] ; output: corresponding symbol
[(:+ digit) ; input: digits
(token-NUMBER (string->number lexeme))])) ; outout: a NUMBER token whose value is
; ; the number
; ; note: (token-value token)
; returns the number
;;; TEST
(define input (open-input-string "123+456"))
(lex input) ; (position-token (token 'NUMBER 123) (position 1 #f #f) (position 4 #f #f))
(lex input) ; (position-token '+ (position 4 #f #f) (position 5 #f #f))
(lex input) ; (position-token (token 'NUMBER 456) (position 5 #f #f) (position 8 #f #f))
(lex input) ; (position-token 'EOF (position 8 #f #f) (position 8 #f #f))
;; Let's make it a little easier to play with the lexer.
(define (string->tokens s)
(port->tokens (open-input-string s)))
(define (port->tokens in)
(define token (lex in))
(if (eq? (position-token-token token) 'EOF)
'()
(cons token (port->tokens in))))
(map position-token-token (string->tokens "123*45/3")) ; strip positions
; Output:
; (list (token 'NUMBER 123)
; '*
; (token 'NUMBER 45)
; '/
; (token 'NUMBER 3))