Commit 44be1f4
committed
Add Cow-based internal tokenizer functions for zero-copy preparation
Add internal _borrowed() functions that return Cow<'a, str> to prepare for
zero-copy tokenization. When the source string needs no transformation
(no escaping), return Cow::Borrowed. When transformation is required,
return Cow::Owned.
The Token enum still uses String, so borrowed values are converted via
to_owned() for now. This maintains API compatibility while preparing the
codebase for a future refactor where Token can hold borrowed strings.
Optimized: comments, quoted strings, dollar-quoted strings, quoted identifiers.
Add safe_slice() helper function for bounds-checked string slicing.1 parent c8acf9f commit 44be1f4
1 file changed
+424
-117
lines changed
0 commit comments