fix(lexer): validate rule names are valid Go identifiers - #468
fix(lexer): validate rule names are valid Go identifiers#468Pastalikek65 wants to merge 1 commit into
Conversation
Rule names are interpolated verbatim into generated codegen function names (match<Lexer><Rule>), so invalid identifiers such as "match:" or ";" produced uncompilable source. Reject them at lexer construction time with a clear error instead. Fixes alecthomas#202.
4947fc5 to
24b31bf
Compare
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24b31bfadf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if rule.Name != "" && !isValidIdent(rule.Name) { | ||
| return nil, fmt.Errorf("lexer: %s.%d: invalid rule name %q: must be a valid Go identifier", key, i, rule.Name) |
There was a problem hiding this comment.
Restricting every rule passed to lexer.New breaks previously valid runtime lexers even when code generation and named grammar references are never used. For example, a lowercase rule named comment-line was accepted and automatically elided by StatefulLexer.Next, while an arbitrary named token can also be consumed through the lexer API or matched by its literal value; after this check, both New and NewSimple reject such definitions solely because the optional generator cannot interpolate the name. Apply this restriction in the generator (or encode generated function names) rather than changing the runtime lexer contract.
Useful? React with 👍 / 👎.
2 participants
Closes #202
Rule names are interpolated verbatim into generated codegen function names (
match<Lexer><Rule>), so invalid identifiers such asmatch:,;,|or@produced uncompilable source (see #202 for the original report).Rather than escaping names in codegen (which would silently rename tokens behind the parser's back), this validates at
lexer.Newconstruction time — the earliest point where a malformed name can be caught, and consistent with @alecthomas's assessment: "they are not valid rule names as only identifiers can be matched by the parser. This is really a validation error."Changes:
lexer/stateful.go: reject non-identifier rule names inNew()with a clear error (lexer: <state>.<idx>: invalid rule name ...)lexer/stateful_test.go: table test covering invalid (match:,;,1two,has space,ünïcode-rune) and valid (Ident,identifier,_private,ünïcode) namesNo behavior change for existing valid lexers; names that were never usable now fail fast instead of emitting broken codegen output.