Skip to content

Various fixes to IDNA code - #1601

Open
roozbehp wants to merge 6 commits into
mainfrom
roozbehp-post-1600
Open

roozbehp wants to merge 6 commits into
mainfrom
roozbehp-post-1600

Conversation

@roozbehp

@roozbehp roozbehp commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Fix IDNA initialization dependencies that made behavior depend on which generator ran first. Eager transforms in Idna could capture ICU normalization before the development-data factory was installed, while unrelated initialization could fail under the NO_PROPS guard.

Also:

  • Use the latest-version unicodetools Normalizer accessors for IDNA processing and test generation.

  • Compute IDNA2008 properties lazily through getInstance(), update the JSP and test callers, and move file-generation setup into the generator.

  • Move IDNA2003’s Unicode 3.2 age-filtered NFKC transform into a lazy holder in StringPrepData. Generalize FilteredUnicodeTransform to accept Transform<String, String> so it can wrap a unicodetools normalizer directly.

  • Define the fixed noncharacter, private-use, and surrogate sets directly, avoiding implicit ICU property lookups during UnicodeProperty initialization.

  • Remove the legacy numeric-pinyin conversion of historical kMandarin values, preserving the property-file values and eliminating eager ICU transliterator initialization.

  • Approver: Feel free to merge on my behalf

    • rebase & merge one or more commits
    • squash & merge multiple commits into one

Remove the global NO_PROPS symbol-table guard from Idna2008 initialization.
Its UnicodeSets already select the development symbol table explicitly;
changing the global table can break ICU transliterator initialization.

Obtain shared IDNA normalization and case-folding transforms directly from
ToolUnicodeTransformFactory. Loading Idna2008 before GenerateIdnaTest must
not permanently capture ICU transforms from the default global factory.
@roozbehp roozbehp mentioned this pull request Oct 9, 2026
2 of 3 tasks
@roozbehp
roozbehp requested review from eggrobin and markusicu October 9, 2026 21:57
@eggrobin

eggrobin commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

I would remove this global guard while retaining the explicit development symbol tables.

I would not. I would rather have the thing crash in some configurations than risk using outdated properties and producing bad data. There is a reason why I added that guard.

@eggrobin

eggrobin commented Oct 9, 2026

Copy link
Copy Markdown
Member

IndexUnicodeProperties initializes. That initialization creates the NumericPinyin-Latin;nfc transliterator, whose rules legitimately query ICU properties.

Nothing about this transliterator seems legitimate anyway. This is from a 15 year old papering over of a change in format of kMandarin, it has no relevance anymore, and we do not try to pretend the format of the other readings did not change.

@eggrobin

eggrobin commented Oct 9, 2026

Copy link
Copy Markdown
Member

we do not try to pretend the format of the other readings did not change

See https://util.unicode.org/UnicodeJsps/character.jsp?a=4E00&history=assigned for many examples.

Comment thread unicodetools/src/main/java/org/unicode/idna/Idna2008.java Outdated
Comment thread unicodetools/src/main/java/org/unicode/idna/Idna.java Outdated
@roozbehp

roozbehp commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Nothing about this transliterator seems legitimate anyway. This is from a 15 year old papering over of a change in format of kMandarin, it has no relevance anymore, and we do not try to pretend the format of the other readings did not change.

Agreed. Dropped.

Comment thread unicodetools/src/main/java/org/unicode/idna/Idna2008.java Outdated
@macchiati

macchiati commented Oct 10, 2026 via email

Copy link
Copy Markdown
Member

Comment thread unicodetools/src/main/java/org/unicode/idna/GenerateIdnaTest.java Outdated
Comment thread unicodetools/src/main/java/org/unicode/idna/Idna2008.java Dismissed
Comment thread unicodetools/src/main/java/org/unicode/idna/Idna2008.java Dismissed
Comment thread unicodetools/src/main/java/org/unicode/idna/Idna2008.java Dismissed
@eggrobin

Copy link
Copy Markdown
Member

… OK, let’s revert #1582, it immediately made itself a nuisance.

@roozbehp
roozbehp requested a review from eggrobin October 10, 2026 01:07
@roozbehp roozbehp changed the title Make IDNA initialization independent of generator order Various fixes to IDNA code Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants