Skip to content

fix(lexer): codegen ? (quest) operator gives back in concatenations - #475

Open
Pastalikek65 wants to merge 1 commit into
alecthomas:masterfrom
Pastalikek65:fix-276-quest-backtracking
Open

fix(lexer): codegen ? (quest) operator gives back in concatenations#475
Pastalikek65 wants to merge 1 commit into
alecthomas:masterfrom
Pastalikek65:fix-276-quest-backtracking

Conversation

@Pastalikek65

Copy link
Copy Markdown

Closes #276

The generated lexer's ? (quest, OpQuest) handling was fully greedy: a concatenation A (X)? B committed to consuming X and never retried the not-taken branch when B subsequently failed. So [A-Z][A-Z][A-Z]?T could not match "EST" — the optional [A-Z] greedily ate the trailing "T", and the literal T then failed. The runtime lexer (Go's regexp engine) backtracks and matched correctly, so generated and runtime lexers diverged.

Fix

cmd/participle/gen_lexer_cmd.go: new emitConcatWithQuest generates a backtracking concatenation whenever a quest appears as a direct sub-expression:

<match head>                        // on failure, propagate -1/goto
altN := p
if np := lX(s, p); np != -1 {       // greedy: take the quest
    p = np
    <match tail>                    // tail failure -> goto fallbackN
    return p
}
fallbackN:
// quest not taken
p = altN
<match tail from altN>
return p
  • Nested quests recurse (labels/vars get unique suffixes via a per-function counter).
  • Every generated closure still returns a single int; backtracking uses only locals and goto, so the zero-allocation token path is preserved.
  • Quest closures that are inlined by concats are no longer emitted (they were previously dead code that failed go build with "declared and not used").

Regression coverage

  • lexer/internal/conformance/conformance_test.go: QuestGiveBack ([A-Z][A-Z][A-Z]?T on "EST" — the exact report) and QuestNumBoth ([\+\-]?([0-9]*\.)?[0-9]+ on 12 -12.5 -.5 +.25), run against both the runtime lexer and a freshly generated one.
  • lexer/internal/basiclexer.go: regenerated via scripts/regen-lexer; the BASIC Number rule exercises two nested quests.

Both conformance peers pass; the stateful BASIC benchmark still yields the same 11101 tokens.

The generated lexer\x27s quest (?, OpQuest) handling was fully greedy: the
concatenation "A (X)? B" committed to consuming X and never retried the
not-taken branch if B then failed. [A-Z][A-Z][A-Z]?T therefore could not
match "EST" (the optional [A-Z] ate the trailing "T"). The runtime lexer
(Go regexp) backtracks and matched fine, so generated and runtime lexers
diverged.

emitConcatWithQuest now generates a backtracking concat for quest
sub-expressions: snapshot the position before the quest, try the greedy
path, and on tail failure goto a fallback label that retries the tail
from the not-taken position. Nested quests recurse; every closure still
returns a single int, preserving the zero-allocation token path.

- gen_lexer_cmd.go: emitConcatWithQuest + pre-pass that suppresses quest
  closures inlined by concats (previously dead code).
- conformance_test.go: quest regression cases ([A-Z][A-Z][A-Z]?T on "EST",
  and the BASIC number [-+]?([0-9]*\.)?[0-9]+ covered by QuestNum).
- basiclexer.go: regenerated.

Fixes alecthomas#276.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ideas for gen lexer supporting "give back" for quest

1 participant