Generate Idna2008.txt

[eggrobin]

This involves some changes to the handling of the one XV8 character (NEW TAI LUE THAM DIGIT ONE), since Idna2008.java was forcing it to be PVALID, with the generators for the mapping table and test file doing more special handling on top of that.

• Approver: Feel free to merge on my behalf
  • rebase & merge one or more commits
  • squash & merge multiple commits into one

[roozbehp]

Shouldn't this be renamed now that it also includes XV8?

[eggrobin]

Well, it doesn’t change. Was it misnamed? Yes. What should it have been named? I don’t know.

[roozbehp]

1. My LLM thinks the new call in MakeUnicodeFiles.java to Idna2008.generateIdna2008() initializes Idna2008 without first selecting the development UCD data, and that without it, Idna2008 uses ICU’s properties. It tried adding a new gc=Lo character to the development UCD data and it says it came out as UNASSIGNED instead of PVALID.

2. It looks like your patch ignores --no-copy. What are the plans for that? Are we dropping it?

[eggrobin]

It looks like your patch ignores --no-copy.

Yes, like the other generators that live outside MakeUnicodeFiles.

What are the plans for that? Are we dropping it?

#1597.

[eggrobin]

Idna2008 uses ICU’s properties

Indeed. (Which is a bit disturbing given that it was used in test generation, though probably not in ways that mattered). Fixed.

[markusicu]

Thank you very much!

[roozbehp]

My LLM is still unhappy about this. Pasting the relevant parts of its output. Pull request to fix at #1601.

1. Generating only Idna2008 now crashes in a fresh JVM.

   I reproduced this through MakeUnicodeFiles with --no-copy --generate '^Idna2008$':

   ExceptionInInitializerError
   Caused by: IllegalArgumentException:
   Don't use any ICU Unicode Properties! Letter=null
   

   The new global NO_PROPS guard at Idna2008.java:35 is installed before IndexUnicodeProperties initializes. That initialization creates the NumericPinyin-Latin;nfc transliterator, whose rules legitimately query ICU properties. The guard blocks that query.

   Preloading IndexUnicodeProperties makes the failure disappear, explaining why other invocation orders can succeed. I would remove this global guard while retaining the explicit development symbol tables.

2. Running Idna2008 first still leaves subsequent UTS46 test generation using ICU’s normalizers.

   The revised calculation directly uses the development normalizer, which fixes Idna2008.txt. However, initializing Idna2008 still initializes its superclass, Idna, whose static final normalizers capture the current ICU factory. Later calls to GenerateIdnaTest.setUnicodeVersion() cannot replace those objects. Uts46 continues using the cached Idna.NFC.

   I reproduced an actual difference in generated test expectations using temporary synthetic UCD entries defining NFC(0378 0301) = 0381:

   Initialization order	ToUnicode result	ToASCII result
   Idna2008 first	0378 0301	xn--lsa1v
   Development environment first	0381	xn--8va

   Both reported no errors. The first result misses the development version’s required NFC normalization step.