fix(lexer): codegen ? (quest) operator gives back in concatenations - #475
Open
Pastalikek65 wants to merge 1 commit into
Open
fix(lexer): codegen ? (quest) operator gives back in concatenations#475Pastalikek65 wants to merge 1 commit into
Pastalikek65 wants to merge 1 commit into
Conversation
The generated lexer\x27s quest (?, OpQuest) handling was fully greedy: the concatenation "A (X)? B" committed to consuming X and never retried the not-taken branch if B then failed. [A-Z][A-Z][A-Z]?T therefore could not match "EST" (the optional [A-Z] ate the trailing "T"). The runtime lexer (Go regexp) backtracks and matched fine, so generated and runtime lexers diverged. emitConcatWithQuest now generates a backtracking concat for quest sub-expressions: snapshot the position before the quest, try the greedy path, and on tail failure goto a fallback label that retries the tail from the not-taken position. Nested quests recurse; every closure still returns a single int, preserving the zero-allocation token path. - gen_lexer_cmd.go: emitConcatWithQuest + pre-pass that suppresses quest closures inlined by concats (previously dead code). - conformance_test.go: quest regression cases ([A-Z][A-Z][A-Z]?T on "EST", and the BASIC number [-+]?([0-9]*\.)?[0-9]+ covered by QuestNum). - basiclexer.go: regenerated. Fixes alecthomas#276.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #276
The generated lexer's
?(quest,OpQuest) handling was fully greedy: a concatenationA (X)? Bcommitted to consumingXand never retried the not-taken branch whenBsubsequently failed. So[A-Z][A-Z][A-Z]?Tcould not match"EST"— the optional[A-Z]greedily ate the trailing"T", and the literalTthen failed. The runtime lexer (Go's regexp engine) backtracks and matched correctly, so generated and runtime lexers diverged.Fix
cmd/participle/gen_lexer_cmd.go: newemitConcatWithQuestgenerates a backtracking concatenation whenever a quest appears as a direct sub-expression:int; backtracking uses only locals andgoto, so the zero-allocation token path is preserved.go buildwith "declared and not used").Regression coverage
lexer/internal/conformance/conformance_test.go:QuestGiveBack([A-Z][A-Z][A-Z]?Ton"EST"— the exact report) andQuestNumBoth([\+\-]?([0-9]*\.)?[0-9]+on12 -12.5 -.5 +.25), run against both the runtime lexer and a freshly generated one.lexer/internal/basiclexer.go: regenerated viascripts/regen-lexer; the BASIC Number rule exercises two nested quests.Both conformance peers pass; the stateful BASIC benchmark still yields the same 11101 tokens.