Skip to content

Commit 44be1f4

Browse files
committed
Add Cow-based internal tokenizer functions for zero-copy preparation
Add internal _borrowed() functions that return Cow<'a, str> to prepare for zero-copy tokenization. When the source string needs no transformation (no escaping), return Cow::Borrowed. When transformation is required, return Cow::Owned. The Token enum still uses String, so borrowed values are converted via to_owned() for now. This maintains API compatibility while preparing the codebase for a future refactor where Token can hold borrowed strings. Optimized: comments, quoted strings, dollar-quoted strings, quoted identifiers. Add safe_slice() helper function for bounds-checked string slicing.
1 parent c8acf9f commit 44be1f4

File tree

1 file changed

+424
-117
lines changed

1 file changed

+424
-117
lines changed

0 commit comments

Comments
 (0)