fix(rtl): only letters vote in RTL text detection - #441
Conversation
The RTL Unicode blocks embed weak-directionality characters — Arabic-Indic digits (U+0660-0669, U+06F0-06F9), number separators (U+066B/U+066C), and combining marks — that are bidi class AN/NSM per UAX #9, not strong RTL. A line containing only Arabic-Indic digits therefore voted RTL (rtl>0, ltr=0) and sort_line_items reversed its item order, while the equivalent ASCII-digit line stayed neutral. Require is_alphabetic() for the RTL vote, mirroring the existing LTR-side rule: only letters vote in either direction. Digit-only and punctuation-only lines now keep their natural left-to-right item order, fixing digit-reversed numbers in Arabic documents.
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
Vowel points like U+064E carry Other_Alphabetic, so is_alphabetic() alone still let a marks-only line vote RTL. Skip combining marks (general category Mn) explicitly via unicode_normalization::char::is_combining_mark, which also covers ccc=0 marks like U+0670 and Thaana vowel signs.
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would auto-approve. Fixes RTL text detection to count only alphabetic, non-combining characters as RTL votes, mirroring the LTR rule, so digit/punctuation-only lines stay neutral instead of reversing order; covered by new tests for Arabic-Indic digits, separators, combining marks, and mixed-letter cases.
Re-trigger cubic
|
The failing Clippy check is pre-existing toolchain drift (CI stable moved to clippy 1.98, which added lints that fire on existing code in main) — fixed separately in #442. This branch's own code passes clippy 1.98 locally. |
Problem
is_rtl_textcounts any character in the RTL Unicode blocks as an RTL vote, but the Arabic blocks embed weak-directionality characters — Arabic-Indic digits (U+0660–0669, U+06F0–06F9), number separators (U+066B/U+066C), and combining marks — that are bidi class AN/NSM per UAX #9, not strong RTL.A line containing only Arabic-Indic digits therefore votes RTL (
rtl > 0, ltr = 0) andsort_line_itemsreverses its item order — producing digit-reversed numbers like0,11instead of11,0— while the equivalent ASCII-digit line stays neutral.Fix
Require
is_alphabetic()for the RTL vote, mirroring the rule the LTR side already had: only letters vote in either direction. Digit-only and punctuation-only lines keep their natural left-to-right order. Divergence is confined to lines with no strong RTL letter; lines with Arabic/Hebrew letters are unaffected.Testing
cargo fmt/cargo clippy -- -D warnings/cargo testall pass.🤖 Generated with Claude Code
Summary by cubic
Only letters vote in RTL text detection, and we now exclude combining marks. Previously any RTL-block character (digits, separators, combining marks) voted RTL and reversed digit-only lines; digit-, punctuation-, and marks-only lines now stay neutral and keep left-to-right order, while lines with Arabic/Hebrew letters remain RTL.
is_rtl_textto count votes only for alphabetic, non-combining chars (viaunicode_normalization::char::is_combining_mark); mirrors the LTR rule and keeps the CJK exclusion.sort_line_itemswhen no strong RTL letters are present.main(clippy 1.98 fixes and toolchain pin); no behavior change.Written for commit e7a3d0f. Summary will update on new commits.