fix(pdf): emit ToUnicode surrogate pairs for non-BMP glyphs (#77) - #78
Merged
Conversation
After #75 added Symbola, supplementary-plane symbols rendered correctly but their ToUnicode bfchar destination was formatted with {cp:X4} — five hex digits for a >U+FFFF codepoint (e.g. <1F6C7>), which is not valid UTF-16BE. Extractors read only the first 16-bit unit, so copy/paste, search, and text extraction returned a truncated, wrong character (U+1F6C7 -> U+1F6C, a Greek letter). BuildToUnicodeCMap now emits the destination via Utf16BeHex, which encodes a BMP codepoint as four hex digits and a supplementary one as its high+low surrogate pair (e.g. U+1F6C7 -> <D83DDEC7>). BMP glyphs are unchanged. Adds PdfToUnicodeTests: Utf16BeHex unit cases (BMP + supplementary) and an end-to-end check that the rendered PDF maps U+1F6C7/U+1F6AB to their surrogate pairs and never the raw 5-hex form, while U+26D4 (BMP) stays a single unit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #77.
Problem
After #75 added the Symbola fallback, supplementary-plane (non-BMP) symbols render correctly, but their ToUnicode mapping is wrong: the
bfchardestination is written as a raw>U+FFFFvalue (e.g.<1F6C7>), which isn't valid UTF-16BE. Extractors read only the first 16-bit unit, so copy/paste, search (Ctrl+F), and text extraction return a truncated, wrong character (U+1F6C7 → U+1F6C, a Greek letter). BMP symbols extract correctly.Root cause
BuildToUnicodeCMapformatted the destination with{cp:X4}— four hex digits for a BMP codepoint, but five for a supplementary one. A ToUnicodebfchardestination must be a UTF-16BE string (16-bit units).Fix
The destination is now emitted via a new
Utf16BeHexhelper (char.ConvertFromUtf32→ per-unitX4):<26D4>) — unchanged.U+1F6C7 → <D83DDEC7>,U+1F6AB → <D83DDEAB>).So non-BMP glyphs are now selectable / searchable / extractable as the correct character, restoring (for the SMP case) the goal of #52.
Tests
Adds
PdfToUnicodeTests:Utf16BeHexunit cases (BMP + supplementary), and an end-to-end check that the rendered PDF maps U+1F6C7/U+1F6AB to their surrogate pairs and never the raw 5-hex form, while U+26D4 (BMP) stays a single 16-bit unit. (The ToUnicode stream is uncompressed, so thebfchardestinations are asserted directly in the PDF bytes.)Full Release suite green (0 failures; +5 tests). Related: #75 (Symbola fallback chain), #72 (non-BMP handling), #52 (embedded fallback + ToUnicode).