Skip to content

Commit ca28f04

Browse files
authored
Merge pull request #587 from derek73/fix/issue-585-nfd-initials
Fix initials dropping the accent from a decomposed (NFD) name
2 parents 10cf228 + 96e8f1b commit ca28f04

14 files changed

Lines changed: 265 additions & 28 deletions

‎docs/design/decisions.md‎

Lines changed: 2 additions & 1 deletion
Large diffs are not rendered by default.

‎docs/design/rules.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2476,6 +2476,10 @@ R3. Rationale: initials abbreviate the person's name words; titles,
24762476
word of its own. A part holding nothing else is a part where it
24772477
is joining nothing, and there it initials like any other name
24782478
word, agreeing with the base.
2479+
The first letter is the one the word's composed spelling begins
2480+
with, written in the form the writer used: a letter written as a
2481+
base letter followed by its combining accent initials whole, and
2482+
so does a syllable written as its separate parts.
24792483
The question is asked of the WHOLE PART, never of a word count. A
24802484
part of two words where one is the connective is still a part
24812485
where it joins — the base of "Jon Dough and" is "Dough and" and
@@ -2500,6 +2504,8 @@ R3. Rationale: initials abbreviate the person's name words; titles,
25002504
"Nguyen, Van Le" → initials="V. L. N."
25012505
"Hassan, Mohamad Ahmad Ali" middle_as_family → initials="M. A. A. H."
25022506
"Sean O'Connor" → initials="S. O." · boundary
2507+
"émile zola" → initials="e\u0301. z."
2508+
"マイケル ジャクソン" → initials="マ. シ\u3099."
25032509
A family that is ALL particles therefore contributes its words
25042510
rather than nothing: they are the base (R2), so they initial.
25052511
"Juan van der" → initials="J. v. d."

‎docs/release_log.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ Release Log
5252

5353
- **Fix case repair breaking a name typed with decomposed accents at each accent.** ``HumanName(unicodedata.normalize("NFD", "josé garcía")).capitalize()`` gives ``José García``, where every release from 1.4.0 through 2.3.0 gave ``José GarcíA``: decomposed text (NFD, which macOS file names and some databases hand back) writes ``í`` as ``i`` followed by a combining accent, and the letters after the accent were repaired as a separate word. A decomposed name now repairs the way its composed spelling does, Mac/Mc names and case-repair masks included, and the output keeps the form it was typed in. See the ``R4`` entry of ``docs/design/decisions.md`` (closes #542)
5454

55+
- **Fix initials dropping the accent from a name typed with decomposed accents.** ``parse(unicodedata.normalize("NFD", "émile zola")).initials()`` gives ``é. z.``, where every release gave ``e. z.``: the initial was a word's first code point, which in decomposed text is the letter without its accent. Decomposed katakana lost its voicing mark the same way and now keeps it: ``マイケル ジャクソン`` typed decomposed initials ``マ. ジ.``, where every release gave ``マ. シ.``. ``HumanName.initials()`` moves alike, a decomposed name's initials stay decomposed, and a composed name's initials do not change. See the ``R3`` entry of ``docs/design/decisions.md`` (closes #585)
56+
5557
- **Change the parse pipeline to copy its state without dataclasses.replace.** Every stage returns a copy of its frozen state, and several also copy tokens one at a time; ``dataclasses.replace`` goes through ``fields()`` and ``__init__`` on every one of those copies. The stages now copy fields directly through a small helper that is limited to the pipeline's own three dataclasses and checks them at import. One parse of the benchmark's reference name makes 36 fewer calls on py3.11 and 3.12 and 54 fewer from 3.13 (on 3.11, ``parse`` 406 to 370 and ``HumanName`` 443 to 407), and the call-count baselines move with them. Recomputable with ``uv run python tools/perf/call_count.py --against e0f1a2f``; the counts for every interpreter are in the ``parse-cost`` entry of ``docs/design/decisions.md``. No user-visible behavior changes (#546)
5658

5759
- **Fix a long given part after a family comma costing quadratic time.** Since 2.3.0, parsing ``"Doe, Jane " + "Smith " * n`` took time growing with the square of the part's length: going from 1,600 to 6,400 words cost 8.4x the time, where 2.2.0 and the comma-less form cost 4x. A run of trailing titles in the same place (``Doe, Jane Smith Prof. Prof. ...``) cost 10x. Both cost 4x again (Python 3.11, measured 2026-09-28). No field moves (closes #553)

‎nameparser/_facade.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -643,7 +643,7 @@ def _process_initial(self, name_part: str,
643643
# #461: the conjunction filter reaches EVERY group; only
644644
# the particle filter is exempted for the given group.
645645
if not conjunction and (firstname or not self._is_particle(word)):
646-
initials.append(word[0])
646+
initials.append(_render._first_letter(word))
647647
if len(initials) > 0:
648648
return self.initials_separator.join(initials)
649649
# Return '' (never empty_attribute_default, which may be None)
@@ -711,7 +711,7 @@ def all_particle_guard(got: list[str], words: list[str]) -> list[str]:
711711
# both surfaces for names whose roles agree. _split_last
712712
# already applies the same guard to the base, which is why
713713
# last_base was never empty here.
714-
return [w[0] for w in words]
714+
return [_render._first_letter(w) for w in words]
715715

716716
def group_initials(groups: list[tuple[Token, ...]],
717717
firstname: bool = False) -> list[str]:

‎nameparser/_render.py‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,39 @@ def render(name: ParsedName, spec: str) -> str:
177177
# rules.md#R3: "initials take the first letter of each given, middle,
178178
# and base family word; titles, suffixes, particles and nicknames
179179
# contribute nothing"
180+
def _first_letter(text: str) -> str:
181+
"""The initial of `text`: its first character as its composed (NFC)
182+
spelling reads it, written in the form `text` was written (#585).
183+
That is the shortest start of `text` that composes to the composed
184+
text's first character -- 'e' + U+0301 for a decomposed 'é', and
185+
the two or three jamo of a decomposed hangul syllable, which are
186+
letters rather than marks -- or, where a mark composing with
187+
nothing stands inside that start, the character decomposed (even
188+
where the writer composed part of it). A text whose first code
189+
point holds a whole character gives text[0], as it always did --
190+
a composed text, and one opening with a character Unicode excludes
191+
from composition -- so a mark that composes with nothing (a
192+
Bengali vowel sign) stays out of the initial in either form. Raises IndexError on an empty `text`, as
193+
text[0] does."""
194+
if text.isascii():
195+
return text[0]
196+
first = unicodedata.normalize("NFC", text)[0]
197+
end = 1
198+
while not unicodedata.normalize("NFC", text[:end]).startswith(first):
199+
end += 1
200+
# end == 1: one code point already holds the first letter, kept
201+
# as written even where NFC would split it (U+0958, excluded from
202+
# composition, is 'क' + a nukta composed; it keeps its nukta here)
203+
if end == 1 or unicodedata.normalize("NFC", text[:end]) == first:
204+
return text[:end]
205+
# A mark that composes with nothing stands, in canonical order,
206+
# between the letter and a mark that does ('o' + U+0331 + U+0301,
207+
# where the composed spelling is 'ó' + U+0331), so no start of the
208+
# text composes to `first` alone: write `first` decomposed, the
209+
# form the writer used for it.
210+
return unicodedata.normalize("NFD", first)
211+
212+
180213
def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str:
181214
"""First letter of each contributing token per group, v1 semantics:
182215
delimiter follows each initial, separator sits between initials
@@ -227,7 +260,7 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str
227260
tokens = (tuple(t for t in tokens if FOLDED_TAG in t.tags)
228261
+ tuple(t for t in tokens if FOLDED_TAG not in t.tags))
229262
values[key] = separator.join(
230-
t.text[0] + delimiter for t in tokens)
263+
_first_letter(t.text) + delimiter for t in tokens)
231264
return _format_spec(spec, values, "initials", _INITIALS_KEYS)
232265

233266

‎tests/v2/test_ledger_guards.py‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3635,8 +3635,11 @@ def _claim(rule: dict) -> _Claim:
36353635
# '田中 太郎 旧姓 佐藤 MA' -- the tolerated row this change
36363636
# added, so radar corpus. Growth into new corpus again; no
36373637
# role joined the list.
3638+
# 2026-10-02, #585: 138 -> 139, one new corpus name and not a
3639+
# wider rule: the decomposed katakana R3 row 'マイケル ジャクソン'
3640+
# (NFD) lies in its script span, and fix(#585) explains it.
36383641
"fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots":
3639-
_Claim(138, ('family', 'given', 'middle'), "397de8444acc", None),
3642+
_Claim(139, ('family', 'given', 'middle'), "0feb71190b77", None),
36403643
# 2026-09-19, #533: 33 -> 68. The count grew with the CORPUS
36413644
# rather than with the rule -- this change added 35
36423645
# maiden-clause names as rules.md example lines and
@@ -4601,6 +4604,8 @@ def _claim(rule: dict) -> _Claim:
46014604
# 'SMITH, VD MA', 'smith, de ma'.
46024605
"fix(#573) a never-given particle opening the given part takes the member behind it":
46034606
_Claim(3, ('family', 'suffix'), "54ce0dda7114", ('DEFAULT',)),
4607+
"fix(#585) a decomposed initial keeps its whole first letter":
4608+
_Claim(2, ('_initials',), "c4f045134f0f", None),
46044609
},
46054610
"expected_since_2.0.0.toml": {
46064611
# The ph removal (#459/#521): one literal name, the cases.py
@@ -4777,8 +4782,11 @@ def _claim(rule: dict) -> _Claim:
47774782
# as the 1.4 twin.
47784783
# 2026-09-19, #533: 137 -> 138, the same one new tolerated
47794784
# corpus name as the 1.4.0 copy, '田中 太郎 旧姓 佐藤 MA'.
4785+
# 2026-10-02, #585: 138 -> 139, one new corpus name and not a
4786+
# wider rule: the decomposed katakana R3 row 'マイケル ジャクソン'
4787+
# (NFD) lies in its script span, and fix(#585) explains it.
47804788
"fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots":
4781-
_Claim(138, ('_ambiguities', 'family', 'given', 'middle'), "397de8444acc", None),
4789+
_Claim(139, ('_ambiguities', 'family', 'given', 'middle'), "0feb71190b77", None),
47824790
# 37 -> 35 with the same 2026-09-05 narrowing as the 1.4 twin,
47834791
# whose entry carries the reason. Here the one name that
47844792
# changed hands, '김민준 박사님', goes to the spaced rule
@@ -5353,6 +5361,8 @@ def _claim(rule: dict) -> _Claim:
53535361
# 'SMITH, VD MA', 'smith, de ma'.
53545362
"fix(#573) a never-given particle opening the given part takes the member behind it":
53555363
_Claim(3, ('_ambiguities',), "54ce0dda7114", ('DEFAULT',)),
5364+
"fix(#585) a decomposed initial keeps its whole first letter":
5365+
_Claim(2, ('_initials',), "c4f045134f0f", None),
53565366
},
53575367
# The 2.3 cycle's first rule, and a facade-only render fix: every
53585368
# role is identical, so `_initials` alone. Reach and digest as in
@@ -5827,6 +5837,8 @@ def _claim(rule: dict) -> _Claim:
58275837
# 'SMITH, VD MA', 'smith, de ma'.
58285838
"fix(#573) a never-given particle opening the given part takes the member behind it":
58295839
_Claim(3, ('_ambiguities',), "54ce0dda7114", ('DEFAULT',)),
5840+
"fix(#585) a decomposed initial keeps its whole first letter":
5841+
_Claim(2, ('_initials',), "c4f045134f0f", None),
58305842
},
58315843
"expected_since_2.1.0.toml": {
58325844
# The ph removal (#459/#521): one literal name, the cases.py
@@ -6546,6 +6558,8 @@ def _claim(rule: dict) -> _Claim:
65466558
# 'SMITH, VD MA', 'smith, de ma'.
65476559
"fix(#573) a never-given particle opening the given part takes the member behind it":
65486560
_Claim(3, ('_ambiguities',), "54ce0dda7114", ('DEFAULT',)),
6561+
"fix(#585) a decomposed initial keeps its whole first letter":
6562+
_Claim(2, ('_initials',), "c4f045134f0f", None),
65496563
},
65506564
"expected_since_2.3.0.toml": {
65516565
# The ph removal (#459/#521): one literal name, the cases.py
@@ -6865,6 +6879,8 @@ def _claim(rule: dict) -> _Claim:
68656879
# 'SMITH, VD MA', 'smith, de ma'.
68666880
"fix(#573) a never-given particle opening the given part takes the member behind it":
68676881
_Claim(3, ('_ambiguities',), "54ce0dda7114", ('DEFAULT',)),
6882+
"fix(#585) a decomposed initial keeps its whole first letter":
6883+
_Claim(2, ('_initials',), "c4f045134f0f", None),
68686884
},
68696885
}
68706886

‎tests/v2/test_properties.py‎

Lines changed: 48 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -3106,12 +3106,12 @@ def test_the_case_only_walk_can_fail(
31063106
assert any("'Ph.D.' -> 'PhD'" in line for line in failures), failures
31073107

31083108

3109-
# --- #542: a decomposed name repairs as its composed twin does -------
3109+
# --- #542/#585: a decomposed name renders as its composed twin does -
31103110
# The same texts as the walk above, those that decompose at all, each
3111-
# parsed and repaired in both forms and compared once the decomposed
3112-
# output is composed again. A text whose two forms PARSE differently
3113-
# is set aside rather than compared: repair follows the parse, so the
3114-
# comparison would report the parse, and the one class that does so is
3111+
# parsed in both forms, repaired and initialled, and compared once the
3112+
# decomposed output is composed again. A text whose two forms PARSE
3113+
# differently is set aside rather than compared: both views follow the
3114+
# parse, so the comparison would report the parse, and the one class that does so is
31153115
# unspaced hangul, which segmentation matches as written by decision
31163116
# (docs/usage.rst, "Decomposed text"). The test below pins that filter.
31173117

@@ -3120,7 +3120,7 @@ def _hangul(text: str) -> bool:
31203120
for c in unicodedata.normalize("NFC", text))
31213121

31223122

3123-
def _nfd_repair_findings() -> tuple[list[str], list[str], int]:
3123+
def _nfd_twin_findings() -> tuple[list[str], list[str], int]:
31243124
"""(disagreements, texts set aside, texts compared)."""
31253125
def nfc(text: str) -> str:
31263126
return unicodedata.normalize("NFC", text)
@@ -3139,6 +3139,14 @@ def nfc(text: str) -> str:
31393139
set_aside.append(text)
31403140
continue
31413141
compared += 1
3142+
human, twin = HumanName(decomposed), HumanName(nfc(text))
3143+
for surface, decomposed_initials, composed_initials in (
3144+
("core", name.initials(), composed.initials()),
3145+
("facade", human.initials(), twin.initials())):
3146+
if nfc(decomposed_initials) != composed_initials:
3147+
out.append(f"[initials, {surface}] {text!r}: "
3148+
f"{nfc(decomposed_initials)!r} != "
3149+
f"{composed_initials!r}")
31423150
for force in (False, True):
31433151
got = [nfc(t.text) for t in
31443152
parser.capitalized(name, force=force).tokens]
@@ -3147,7 +3155,6 @@ def nfc(text: str) -> str:
31473155
if got != want:
31483156
out.append(f"[core, force={force}] {text!r}: "
31493157
f"{got!r} != {want!r}")
3150-
human, twin = HumanName(decomposed), HumanName(nfc(text))
31513158
for force in (False, True):
31523159
human.capitalize(force=force)
31533160
twin.capitalize(force=force)
@@ -3159,25 +3166,31 @@ def nfc(text: str) -> str:
31593166
return out, set_aside, compared
31603167

31613168

3162-
def test_a_decomposed_name_repairs_as_its_composed_twin() -> None:
3163-
"""rules.md#R4 read as an invariant over encodings: a letter written
3164-
as a base letter and its combining accent repairs as the letter
3165-
written whole does (#542). Core and facade, plain and forced. The output is compared composed
3166-
because repair keeps the form it was given, which the case-only
3167-
walk above holds separately.
3168-
3169-
Recorded negative control, measured 2026-10-02 with 97af1f02's
3170-
_render.py, before a word ran on through its marks, over this
3171-
change's corpus (its own two R4 rows included): 144 disagreements
3172-
over the 137 texts compared, 47 more set aside, every one hangul
3173-
('JOSÉ GARCÍA' repairing to 'José GarcíA'). The live control is the test below."""
3174-
failures, set_aside, compared = _nfd_repair_findings()
3169+
def test_a_decomposed_name_renders_as_its_composed_twin() -> None:
3170+
"""rules.md#R3 and #R4 read as an invariant over encodings: a
3171+
letter written as a base letter and its combining accent initials
3172+
(#585) and repairs (#542) as the letter written whole does. Core
3173+
and facade; repair plain and forced. The output is compared
3174+
composed because both views keep the form they were given.
3175+
3176+
Recorded negative controls, measured 2026-10-02. Repair, with
3177+
97af1f02's _render.py, before a word ran on through its marks, over
3178+
#542's corpus (its own two R4 rows included): 144 disagreements over
3179+
the 137 texts compared, 47 more set aside, every one hangul
3180+
('JOSÉ GARCÍA' repairing to 'José GarcíA'). Initials, with
3181+
_first_letter taking text[0] again, over #585's corpus (its own two
3182+
R3 rows included, so 139 texts compared where #542's had 137): 44
3183+
disagreements, 22 texts each failing on both surfaces, katakana as
3184+
well as Latin ('マイケル ジャクソン' initialling 'マ. シ.', the
3185+
dakuten of 'ジ' decomposing to a combining mark). The live
3186+
controls are the two tests below."""
3187+
failures, set_aside, compared = _nfd_twin_findings()
31753188
assert compared, "no decomposable text was compared"
31763189
assert all(_hangul(text) for text in set_aside), (
31773190
"a non-hangul text parses differently decomposed: "
31783191
f"{[t for t in set_aside if not _hangul(t)]}")
31793192
assert not failures, (
3180-
f"{len(failures)} decomposed repair(s) disagree:\n"
3193+
f"{len(failures)} decomposed rendering(s) disagree:\n"
31813194
+ "\n".join(failures[:10]))
31823195

31833196

@@ -3188,10 +3201,23 @@ def test_the_decomposed_walk_can_fail(
31883201
import nameparser._render as render_module
31893202
monkeypatch.setattr(render_module, "_past_marks",
31903203
lambda text, end: end)
3191-
failures, _, _ = _nfd_repair_findings()
3204+
failures, _, _ = _nfd_twin_findings()
31923205
assert any("'José', 'GarcíA'" in line for line in failures), failures
31933206

31943207

3208+
def test_the_decomposed_initials_walk_can_fail(
3209+
monkeypatch: pytest.MonkeyPatch) -> None:
3210+
"""Taking a token's first code point again -- #585's defect -- has
3211+
to be seen by the walk above, on both surfaces."""
3212+
import nameparser._render as render_module
3213+
monkeypatch.setattr(render_module, "_first_letter",
3214+
lambda text: text[0])
3215+
failures, _, _ = _nfd_twin_findings()
3216+
for surface in ("core", "facade"):
3217+
assert any(line.startswith(f"[initials, {surface}]")
3218+
for line in failures), failures
3219+
3220+
31953221
def test_a_delimiter_core_reads_as_if_it_were_not_written() -> None:
31963222
"""#538: under a core-bearing policy, a maiden clause reads exactly
31973223
as the same text written without the core. The core is structure

0 commit comments

Comments
 (0)