Skip to content

fix(rtl): only letters vote in RTL text detection - #441

Open
abimaelmartell wants to merge 3 commits into
mainfrom
fix/rtl-weak-chars-vote
Open

fix(rtl): only letters vote in RTL text detection#441
abimaelmartell wants to merge 3 commits into
mainfrom
fix/rtl-weak-chars-vote

Conversation

@abimaelmartell

@abimaelmartell abimaelmartell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Problem

is_rtl_text counts any character in the RTL Unicode blocks as an RTL vote, but the Arabic blocks embed weak-directionality characters — Arabic-Indic digits (U+0660–0669, U+06F0–06F9), number separators (U+066B/U+066C), and combining marks — that are bidi class AN/NSM per UAX #9, not strong RTL.

A line containing only Arabic-Indic digits therefore votes RTL (rtl > 0, ltr = 0) and sort_line_items reverses its item order — producing digit-reversed numbers like 0,11 instead of 11,0 — while the equivalent ASCII-digit line stays neutral.

Fix

Require is_alphabetic() for the RTL vote, mirroring the rule the LTR side already had: only letters vote in either direction. Digit-only and punctuation-only lines keep their natural left-to-right order. Divergence is confined to lines with no strong RTL letter; lines with Arabic/Hebrew letters are unaffected.

Testing

  • New unit tests: Arabic-Indic digit-only lines (plain, extended, and with number separators) are not RTL; Arabic letters + Arabic-Indic digits still RTL; letters + combining marks still RTL.
  • cargo fmt / cargo clippy -- -D warnings / cargo test all pass.
  • Full-corpus A/B against a main-built binary: only Arabic-content documents change, restoring correct digit order in numeric fragments (e.g. a temperature-gradient table now reads 11,0 / 20,0 / 32,0 instead of digit-reversed values).

🤖 Generated with Claude Code


Summary by cubic

Only letters vote in RTL text detection, and we now exclude combining marks. Previously any RTL-block character (digits, separators, combining marks) voted RTL and reversed digit-only lines; digit-, punctuation-, and marks-only lines now stay neutral and keep left-to-right order, while lines with Arabic/Hebrew letters remain RTL.

  • Restricts is_rtl_text to count votes only for alphabetic, non-combining chars (via unicode_normalization::char::is_combining_mark); mirrors the LTR rule and keeps the CJK exclusion.
  • Adds tests for Arabic-Indic digits, separators, combining marks, and mixed-letter cases; effect is limited to sort_line_items when no strong RTL letters are present.
  • Merges main (clippy 1.98 fixes and toolchain pin); no behavior change.

Written for commit e7a3d0f. Summary will update on new commits.

Review in cubic

The RTL Unicode blocks embed weak-directionality characters — Arabic-Indic
digits (U+0660-0669, U+06F0-06F9), number separators (U+066B/U+066C), and
combining marks — that are bidi class AN/NSM per UAX #9, not strong RTL.
A line containing only Arabic-Indic digits therefore voted RTL (rtl>0,
ltr=0) and sort_line_items reversed its item order, while the equivalent
ASCII-digit line stayed neutral.

Require is_alphabetic() for the RTL vote, mirroring the existing LTR-side
rule: only letters vote in either direction. Digit-only and punctuation-only
lines now keep their natural left-to-right item order, fixing digit-reversed
numbers in Arabic documents.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic

Comment thread src/text_utils.rs Outdated
Vowel points like U+064E carry Other_Alphabetic, so is_alphabetic() alone
still let a marks-only line vote RTL. Skip combining marks (general
category Mn) explicitly via unicode_normalization::char::is_combining_mark,
which also covers ccc=0 marks like U+0670 and Thaana vowel signs.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Shadow auto-approve: would auto-approve. Fixes RTL text detection to count only alphabetic, non-combining characters as RTL votes, mirroring the LTR rule, so digit/punctuation-only lines stay neutral instead of reversing order; covered by new tests for Arabic-Indic digits, separators, combining marks, and mixed-letter cases.

Re-trigger cubic

@abimaelmartell

Copy link
Copy Markdown
Member Author

The failing Clippy check is pre-existing toolchain drift (CI stable moved to clippy 1.98, which added lints that fire on existing code in main) — fixed separately in #442. This branch's own code passes clippy 1.98 locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant