Skip to content

fix: include token type in unexpected token error messages - #463

Open
Pastalikek65 wants to merge 2 commits into
alecthomas:masterfrom
Pastalikek65:improve-unexpected-token-error
Open

fix: include token type in unexpected token error messages#463
Pastalikek65 wants to merge 2 commits into
alecthomas:masterfrom
Pastalikek65:improve-unexpected-token-error

Conversation

@Pastalikek65

Copy link
Copy Markdown

Unexpected token errors now report the symbolic name of the lexer's token type when one exists, e.g.:

unexpected token "group" of type Keyword (expected <ident>)

Token types without a symbolic name (such as literals) keep the previous format, e.g.:

unexpected token ")" (expected <ident>)

This is the format suggested by @alecthomas in #265 ("unexpected token "group" of type keyword").

Implementation notes:

  • parseContext now carries the lexer's symbol table (built once from the parser's lexer.Definition); every UnexpectedTokenError construction point (7 sites across parser.go/nodes.go) fills the new TokenType field.
  • UnexpectedTokenError.TokenType is ordered after the pre-existing exported fields, so external positional struct literals remain source-compatible.

Verification: go test ./... (root + _examples), golangci-lint run — no new issues vs. baseline.

Fixes #265.

Unexpected token errors now include the symbolic name of the lexer token type when one exists, e.g. "unexpected token \"group\" of type Keyword". Token types without a symbolic name (such as literals) are unchanged.

Fixes alecthomas#265.

@alecthomas alecthomas left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm. I think this should use the same <lower-symbol> syntax that we already use elsewhere for grammars.

@Pastalikek65

Copy link
Copy Markdown
Author

Thanks @alecthomas! I've updated the format to use the same <lower-symbol> syntax as elsewhere in the grammar (e.g. unexpected token "struct" of type <ident>), via tokenTypeName returning <lower>-wrapped symbols. All tests updated and passing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Include token type in unexpected token error message

2 participants