How to learn lexy (more advanced concepts like position in custom callbacks)

[totalgee]

Hi there, I'm having a hard time wrapping my mind around more complex productions with lexy. In some cases, I want to do things like specify a complex rule and do the parsing myself, using a callback (and knowing where various things occurred in parsing, using dsl::position.

Besides the website documentation (which is great) -- are there any other recommended resources on how to work one's way up in expertise with lexy? I've gone through the docs, and tried many examples, but I find the playground lacking because it only seems to handle the parsing part (rule) and not the value part, which is where I seem to struggle most.

I guess what I'm asking for specifically would be an example of using dsl::position (possibly more than once in a rule) and a return values, using a custom callback that takes those positions as arguments (along with other parsed things, like ints or strings). Does one just use char const* to take parsed position arguments? Or lexemes, or input_location? I've not managed to make an example that compiles... And an example for dsl::position like this one from the Reference doesn't include a value definition.

Thanks!

[foonathan]

Besides the website documentation (which is great) -- are there any other recommended resources on how to work one's way up in expertise with lexy? I've gone through the docs, and tried many examples, but I find the playground lacking because it only seems to handle the parsing part (rule) and not the value part, which is where I seem to struggle most.

Right, the playground uses the lexy::parse_as_tree or lexy::trace action which ignore the value callbacks. Supporting value callbacks there is probably not worth it, as you would have to bring in your own type definitions and stuff.

I unfortunately do not have any other resources besides the documentation.

Does one just use char const* to take parsed position arguments?

Yes, dsl::position matches nothing and just produces an iterator to the current parsing position. If you are parsing a contiguous input with the lexy::default_encoding that is a char const*. For lexy::utf8_encoding it is a char8_t const*.

Or lexemes, or input_location?

lexy::lexeme is produced by dsl::capture (and some other rules): https://lexy.foonathan.net/reference/dsl/capture/#capture

lexy::input_location is only used if you want to turn a const char* into line, column. For that you use lexy::get_input_location: https://lexy.foonathan.net/reference/input_location/#get_input_location

I've not managed to make an example that compiles...

Can you post the error messages you got? Unless you're using MSVC, it should have told you the relevant information.

And an example for dsl::position like this one from the Reference doesn't include a value definition.

Good catch, I will update it to use this example: https://godbolt.org/z/MdK77ErqM

[totalgee]

For what I was trying to do, I think I was approaching it the wrong way. I was wanting to do my own parsing of a more complicated thing (multiple tokens with different possibilities). I changed my approach, breaking the possibilities down into distinct sub-productions and then using "or" alternatives (|) with them, and was eventually able to get things working. I still find the Lexy compile errors pretty hard to parse, they don't give me much help to figure out where/what to fix. But gradually I'm getting a better intuition for it... ;-)

@foonathan Thanks again for the tips, reading your answers and then struggling a bit more convinced me to try a different approach, which worked. I still don't know how to do what I had originally intended, where I wanted to parse a set of multiple tokens/values with dsl::positions before/after the whole set, and I wanted the text between these "positions" as text (std::string_view or std::string) in a callback, where I could output whatever resulting value I wanted. But this is not important (to me) right now, just a curiosity if this was even possible.

[foonathan]

I still find the Lexy compile errors pretty hard to parse, they don't give me much help to figure out where/what to fix.

Feel free to submit issues if you can't understand an error message. I consider it a failure in design.