Triggered by discussion in #158286 @rust-lang/libs-api concluded that our debug escapes are escaping too many characters and are not internationalization-friendly.
E.g. ຄົນສົ່ງໜັງສື currently gets escaped to ຄ\u{ebb}ນສ\u{ebb}\u{ec8}ງໜ\u{eb1}ງສ\u{eb7}
We don't have precise rules ourselves what should and shouldn't be escaped and we'd like to gather input and help from unicode/internationalization experts to find a better balance between various debugging goals and learn what the tradeoffs are.
Once we better understand the tradeoffs and have decided on an approach implementation help, testdata and similar would also be appreciated.
Some desiderata that were brought up during discussion:
- living languages¹ should not be excessively mangled by escaping and remain readable.
A few ligatures getting torn per sentence probably is ok, every other letter getting replaced by an escape wouldn't be
- surprising/unexpected/unprintable characters - e.g. a ZWJ or BOM in the middle of ascii strings/latin script - should get escaped
- debug printing shouldn't mess up terminals → control sequences and directionality overrides should remain escaped
- binary size might also be a concern if additional unicode tables are required
¹ "Any text one could pluck from a random international wikipedia page that isn't about ancient typography"
Triggered by discussion in #158286 @rust-lang/libs-api concluded that our debug escapes are escaping too many characters and are not internationalization-friendly.
E.g.
ຄົນສົ່ງໜັງສືcurrently gets escaped toຄ\u{ebb}ນສ\u{ebb}\u{ec8}ງໜ\u{eb1}ງສ\u{eb7}We don't have precise rules ourselves what should and shouldn't be escaped and we'd like to gather input and help from unicode/internationalization experts to find a better balance between various debugging goals and learn what the tradeoffs are.
Once we better understand the tradeoffs and have decided on an approach implementation help, testdata and similar would also be appreciated.
Some desiderata that were brought up during discussion:
A few ligatures getting torn per sentence probably is ok, every other letter getting replaced by an escape wouldn't be
¹ "Any text one could pluck from a random international wikipedia page that isn't about ancient typography"