Self-contained parser
Object improvements
- dict allows to use any immutable Tokay value as keys
- Enforcement of void paradigm
Syntax
- Inline sequences
- Item retrieval and assignment
- Char-builtin replaces [...]-syntax and .
- Area syntax @(...)
Changelog
2023-01-13
Release of Tokay v0.6

After seven months of intense development, we're happy to announce version 0.6 of Tokay as another milestone towards a stable and final version of the language.

Self-contained parser

One of the main goals of this release was to make tokay.tok become Tokay's main grammar. This eliminates the maintenance of two parsers (parser.rs and tokay.tok), and clears the way for extensive future changes on the grammar and AST.

parser.rs, which is Tokay's internal parser, is now generated from a version of tokay.tok parsing tokay.tok executed by Tokay. It's quite like a lizard eating its own tail, but yeah, it is!

This made the entire macro-based compiler, with a grammar expressed as Rust macro callls, obsolete.

Object improvements

Plenty of improvements went into the object system.

dict allows to use any immutable Tokay value as keys

Most values in Tokay have been declared as immutable - the only mutable values are dict and list, as in this case the content of the value object is modified, rather than replaced (as it is the case with atomic values).

This finally made it possible to use any Tokay value - except list and dict - as keys for dict as well, and not only str.

d = dict()
d[true] = 1
d[42] = true
d[23.5] = "yes"

Enforcement of void paradigm

The void-value defines nothing and can be used so. To remove an item from d above, just set it to void:

d[true] = void

In case a defined "nothing" should be set, use null.

Syntax

The syntax was enhanced as well. Tokay v0.6 also contains a syntax draft for the upcoming generic parselets feature. The language accepts syntax, which might raise a todo!()-call and stop.

Inline sequences

Within brackets (...), Tokay now accepts for inline-sequences in an improved version. () defines the empty list.

Other sequences either result in a list or in a dict, depending on their content. This behavior should be changed im Tokay v0.7, so that lists can be defined explicitly, and everything else is a dict.

Item retrieval and assignment

The retrieval of items in objects has been standardized. This is done using the methods *_get_item and *_set_item*.

l = (1, 2, 3)
l[0]  # 1
l[1] = 4

d = (a => 1, b => 2)
d["a"]  # 1
d["c"] = 3  # inserts 3 as "c"

The current implementations are

str objects cannot be modified,

Char-builtin replaces [...]-syntax and .

The previous [...]-syntax for character-classes was removed as it is reserved for the upcoming list syntax, and was replaced by the Char<>-builtin.

Char         # previously `.`
Char<a-z>    # previously `[a-z]`
Char<^a-z>+  # previosuly `[^a-z]+`

The new syntax already follows the planned syntax design principle for generic parselets, but is integrated into the language and yields in Char-tokens.

Area syntax @(...)

This new syntax allows to define a parseable sequences with in-place reader extend.

Given the program

x = (Int ',' Int | Ident)
print(repr(x))

the print will never be executed, because the sequence in the assignment matches input, and Tokay's default behavor in a block is to accept a matching sequence as an alternative.

To avoid this behavior (which, indeed, might be a wanted behavior!) and without introducting a new parselet to resolve this, the area syntax can be used:

x = @(Int ',' Int | Ident)
print(repr(x))

The region inside the @(...) gets parsed separately, and the reader is extended so that the program continues as expected.

Changelog

This is the full changelog of this version.