[vera-rubin]

Hangul decomposition skipped grapheme markers and boundary state updates with CHARBOUND. We now process each Jamo through the shared boundary handling only with CHARBOUND, preserving the original fast path otherwise. We kept all Hangul boundary regressions, added byte checks for normalization without CHARBOUND, and make check passes.

Fixes #373

— ChipLab

[stevengj]

I don't speak Korean, but it looks like utf8proc_decompose_char is just working around the problem rather than fixing the underlying issue with grapheme-break markers and Hangul decomposition. Could you explain why the decomposition can't be fixed directly?

[vera-rubin]

You're right, the decomposition can be fixed directly. The Hangul shortcut wrote Jamo straight to the output, skipping the shared boundary handling and leaving its state unchanged. This revision processes each Jamo through utf8proc_decompose_char, just like table-based decompositions, so it uses the common marker and state handling.

— ChipLab

[stevengj]

LGTM. A little ugly to duplicate code here, but better than slowing down the common decomposition path for an option that probably few people use. cc @eschnett, what do you think?

[stevengj]

@vera-rubin, now that we are splitting the CHARBOUND branch into a slow path, it's better to optimize the slow path for simplicity. I seem to recall that your original version of this patch was simpler (but slower) with paired recursive calls to utf8proc_decompose_char? Could you remind me of your initial implementation so that we can compare?

Note, by the way, that we will squash the commits when we merge, so don't worry about minimizing the number of commits — in the future, I would rather you didn't force-push here, since that means I can no longer see previous versions of this PR.

[vera-rubin]

Sure, the earlier version is f4a61c9. It ran each Jamo back through utf8proc_decompose_char in a short loop (the same loop the CHARBOUND slow path uses), so no boundary logic was duplicated. Noted on force-pushes: we will add commits from here on.

— ChipLab

[stevengj]

No, I mean your initial PR version, which was the version before f4a61c9 (before this comment: #374 (comment))

[vera-rubin]

Sorry, wrong hash. The initial version is d6b7835: under CHARBOUND it decomposed the syllable into Jamo with one utf8proc_decompose_char call without CHARBOUND, then passed each Jamo back through utf8proc_decompose_char with CHARBOUND.

— ChipLab

[stevengj]

Let's revert to d6b7835, which is shorter and involves less code duplication, even if it sacrifices some efficiency in the (rarer) CHARBOUND case.

[vera-rubin]

Done: 616e745 restores that version of the CHARBOUND path as a new commit on top. The regression tests are unchanged.

— ChipLab