Appendix D: Builtins

Functions

Tokens

The following tokens are built into Tokay and can be used immediatelly. Programs can override these constants on-demand.

Token Token+ Description
AlphabeticAlphabeticsAll Unicode characters having the Alphabetic property
AlphanumericAlphanumericsThe union of Alphabetic and Numeric
Any / .-Any character, except EOF
AsciiAsciisAll characters within the ASCII range.
AsciiAlphabeticAsciiAlphabeticsAll ASCII alphabetic characters [A-Za-z]
AsciiAlphanumericAsciiAlphanumericsASCII alphanumeric characters [0-9A-Za-z]
AsciiControlAsciiControlsAll ASCII control characters [\x00-\x1F\x7f]. SPACE is not a control character.
AsciiDigitAsciiDigitsASCII decimal digits [0-9]
AsciiGraphicAsciiGraphicsASCII graphic character [!-~]
AsciiHexdigitAsciiHexdigitsASCII hex digits [0-9A-Fa-f]
AsciiLowercaseAsciiLowercasesAll ASCII lowercase characters [a-z]
AsciiPunctuationAsciiPunctuationsAll ASCII punctuation characters [-!"#$%&'()*+,./:;<=>?@[\\\]^_`{|}~]
AsciiUppercaseAsciiUppercasesAll ASCII uppercase characters [A-Z]
AsciiWhitespaceAsciiWhitespacesAll characters defining ASCII whitespace [ \t\n\f\r]
ControlControlsAll Unicode characters in the controls category
DigitDigitsASCII decimal digits [0-9]
EOF-Matches End-Of-File.
LowercaseLowercasesAll Unicode characters having the Lowercase property
NumericNumericsAll Unicode characters in the numbers category
UppercaseUppercasesAll Unicode characters having the Uppercase property
WhitespaceWhitespacesAll Unicode characters having the White_Space property
Void-The empty token, which consuming nothing, but consumes!

The respective properties of the built-in character classes is described in Chapter 4 (Character Properties) of the Unicode Standard and specified in the Unicode Character Database in DerivedCoreProperties.txt.