Switch from pest to antlr4rust?

[wackywendell]

Summary

The Rust ANTLR ecosystem is currently undergoing a major consolidation. With substrait-validator recently migrating to antlr4rust, it may be worth considering whether substrait-explain should follow suit to ensure long-term maintainability, performance, and alignment with modern ANTLR versions (4.13.2+).

Context

Currently, the Rust target for ANTLR is split between the legacy antlr-rust crate (largely unmaintained) and the more modern antlr4rust. There is an active effort to upstream the latter and consolidate both under the original antlr-rust name. Given that substrait-explain relies on a generated parser, moving to the community-preferred toolchain could resolve existing technical debt and prevent future compatibility issues.

Pros of Migrating

• Modern ANTLR Support: antlr4rust supports ANTLR 4.13.2+, allowing the use of newer grammar features and bug fixes not available in the legacy 4.8-based targets.
• Ecosystem Alignment: Aligning with substrait-validator would make it easier to share grammar improvements and logic between the two projects. Similarly, the main Substrait repo has an ANTLR grammar.
• Active Maintenance: The current maintainers of antlr4rust are actively working with the official ANTLR team to make Rust an official target.

Cons & Complications

• Migration Effort: The API for the generated visitor and listener patterns differs between the legacy and modern crates, requiring a non-trivial refactor of the current parsing logic in src/.
• Crate Renaming Flux: The community is currently in the process of renaming antlr4rust to antlr-rust. Implementing this change now might require a second update to the Cargo.toml once the consolidation is finalized.
• Build Tooling: Both ANTLR and the antlr4rust projects would need to be installed at build-time, increasing dependencies, and making the build/CI setup more complicated.

Suggestion

Let's perhaps wait for:

1. The antlr-rust / antlr4rust naming discussion / consolidation: Crate to upstream antlr4rust/antlr4#24
2. Some resolution of the relationship with the official ANTLR distribution: [Feature Request] Target for Rust Programming Language antlr/antlr4#1839