Skip to content

Commit 0aefefe

Browse files
derek73claude
andcommitted
Fix what the review of the fix commit found (#596)
- 김゙민준 moves in its report at 2.0.0 too, not only 2.1.0/2.2.0: the #449 rule joins the 2.0.0 ledger (seven #449 rules there now), and decisions.md's MEASURED sentence says so; 1.4.0 has no report and is parity. - The hangul row is tolerated: rules.md's W Background leaves a mark after a non-kana unpromised, and a contract row contradicted it. It moves to the radar corpus; decisions.md says "watches", not "pins". - Stale "follows a character" in the three fix(#596) ledger comments and "every voicing mark" in _policy.py now say "after a kana". - The 147 -> 153 native-script claim bumps carry dated comments, and the new #449 rule has its _MUST_NOT_MATCH near-misses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 2ca2263 commit 0aefefe

10 files changed

Lines changed: 46 additions & 11 deletions

‎docs/design/decisions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -469,7 +469,7 @@ Declined:
469469
- 2026-07-27 (script-scoped order amendment) — the family-first override is keyed to the SCRIPT of the written name, never to a guessed language: wholly-Han, wholly-Hangul, and kana-licensed Japanese read family-first because zh, ko and ja all write family-first in native script; wholly-katakana names are predominantly transcriptions and keep the declared order. Latin transliterations are never touched.
470470
- 2026-07-29 #272 — the kana license: Han∪kana with at least one kana cannot be Chinese and is not a transcription, so 高橋みなみ reads family-first though it is written in two scripts.
471471
- 2026-10-03 #594 — HALFWIDTH KATAKANA IS KATAKANA. The script table classifies the halfwidth kana block U+FF65–FF9F as `Script.KATAKANA`, so `山田 タロウ` reads family 山田 as `山田 エミ` does; every release through 2.3.0 left the block unclassified and read it given 山田, family タロウ, the positional reading. Legacy data written for JIS X 0201 systems, which had no kanji, spells names this way — bank, payroll and CSV exports. DECIDED by widening the table rather than by folding: nothing rewrites the text before parsing (rules.md's T Background), so normalization at tokenize was never available, and an NFKC fold at classification — the other place the issue offered — reaches far past the kana (`JOHN` to `JOHN`, `㈱` to `(株)`), and it would move a wholly-katakana name across W4's line: a voicing mark with no precomposed form to join (`ア゙`, `ン゙`) folds to the COMBINING U+3099, which sits in the HIRAGANA block, so the folded token reads as kana-licensed Japanese and `ア゙イ タロウ` would read family-first where the unfolded one keeps the declared order (measured 2026-10-03 by parsing the NFKC spelling `ア゙イ タロウ`, which reads family `ア゙イ`; a two-input sweep of halfwidth names against their NFC(NFKC) spellings differed only where such a mark stood; RECOMPUTE: parse each halfwidth name and `unicodedata.normalize("NFC", unicodedata.normalize("NFKC", name))` and compare the fields). The interpunct flank guard is NOT the hazard, though an earlier wording of this bullet said it was: it reads raw text and asks only whether a character is classified at all, which hiragana and katakana both are. The same measurement shows a pre-existing gap the fold would have widened rather than created: full-width katakana typed with an uncomposable combining U+3099 already takes the kana license today. A block range is how the table already decides (U+30FB and the voicing marks are in by block), and it takes the voicing marks U+FF9E/U+FF9F in because they are spacing characters following their base. The span starts at U+FF65, the halfwidth nakaguro, which tokenize already treats as a separator (rules.md#T2), and stops short of U+FF61–FF64, whose U+FF61 is a full stop. Everything keyed on the table follows: the kana license, the second-East-Asian-word count that keeps a segmenter from being asked (`高橋一郎 タロウ`), the 间隔号's flank guard (`タロウ·ヤマダ` divides), and `_NO_INITIALS`, which closes the limit #cjk-full-stops recorded (`タナカ. John` read title `タナカ.`). THE RATIONALE FOR WHOLLY-KATAKANA NAMES IS AMENDED, NOT THE BEHAVIOR: a wholly-halfwidth name keeps the declared order exactly as a wholly-katakana one does (`ヤマダ タロウ` reads given ヤマダ), but the 2026-07-27 reason — that such names are predominantly transcriptions — is weaker here, since a system with no kanji wrote Japanese names in katakana too, family-first. What stands is the narrower claim: the script cannot tell a Japanese reading from a transcription, so it settles no order, and a caller who knows their katakana is Japanese adds `(Script.KATAKANA, FAMILY_FIRST)` to `script_orders` (docs/usage.rst's Boundaries section carries the recipe). `name_order=FAMILY_FIRST` works too, at the cost of reversing Latin names as well. MEASURED 2026-10-03: the differential gate saw nothing before this change because no corpus line held halfwidth kana; the eight case rows #594 added are its whole population, five of them contract movers (`山田 タロウ`, `山田 ダイスケ`, `山田 ペーター`, the unspaced `山田タロウ`, `タロウ·ヤマダ`), two tolerated radar movers (`タナカ. John`, `マイケル.`) and one parity row (`ヤマダ タロウ`). RECOMPUTE: parse the eight on this tree and on 2.3.0 from PyPI and read `title` beside the name fields.
472-
- 2026-10-03 #596 — A VOICING MARK TAKES THE SCRIPT OF THE KANA BEFORE IT. The four kana voicing marks U+3099–309C are in the hiragana BLOCK, which is how the script table decides, but none belongs to either syllabary: a mark voices the kana it follows. NFC composes one only where a precomposed voiced kana exists (カ + U+3099 is ガ), so `ア゙イ`, `ン゙` and every spacing ゛/゜ kept a hiragana-block character inside a katakana word, the word took the kana license, and a wholly-katakana name read family-first — the gap the #594 entry above measured, present since 2.1.0. DECIDED by deleting, from the classification copy only, every mark standing after a KANA, so the word reads as its base's script; token text is untouched. A mark after anything else — hangul, Han, Latin, or nothing — voices no kana and keeps the table's answer, as before. That scope is a review finding, not a nicety: the first cut deleted every mark with ANY character before it, which handed `김゙민준` the hangul script, and the surname division, which cuts the raw text, then split the mark off its base — given 김, family `゙민준`, unreported, where 2.3.0 left the word whole and reported given-or-family; `parser_for(locales.ZH)` cut `王゙小明` the same way. A parity case row pins `김゙민준`. Declined: taking the marks out of the hiragana range instead. Measured 2026-10-03 on patched copies of this tree, it fixes the katakana names as well but leaves a word holding a mark in NO script, so `あ゙い たろう` loses family-first, `山田 ア゙イ` loses the kana license and reads family `ア゙イ`, and the unspaced `山田ア゙イ` reads given; the two parity case rows #596 added pin those readings. Declined too: narrowing to the two combining marks, which leaves `ア゛イ タロウ` family-first. NFD-typed text was the issue's worry and is not affected by either option, since classification composes first: NFD `やまだ がくと`, `ジョン ガイ` and `김 민준` parse identically under all of them. A mark with nothing before it keeps the table's answer, so `゙アイ タロウ` still reads family-first; that is degenerate input, recorded in rules.md's W Background as unpromised rather than given a branch. ONE ARGUMENT IN THE #594 ENTRY ABOVE NO LONGER HOLDS: an NFKC fold would no longer move a wholly-katakana name across W4's line, since the combining mark it produces (`ア゙` to `ア゙`) now reads as the katakana it follows — #594's own RECOMPUTE, run 2026-10-03 on this tree over `ア゙イ タロウ`, `ン゙ タロウ`, `山田 ア゙イ`, `マイケル ア゙イ`, `ア゚イ タロウ`, `山田 タロウ` and `ヤマダ タロウ` under all three orders, finds no role differing. The range decision stands on its other ground, the fold's reach past the kana (`JOHN`, `㈱`). The interpunct clause of rules.md#W4 was corrected in the same change: a name the 间隔号 divides keeps the DECLARED order, not its source order — under `name_order=FAMILY_FIRST` `威廉·莎士比亚` reads family 威廉 — and an example pins it. MEASURED 2026-10-03: no corpus line held an uncomposable mark before this change, so the case rows #596 added are the whole population. At 2.1.0, 2.2.0 and 2.3.0 the three `fix(#596)` rows move; at 2.1.0 and 2.2.0 `김゙민준` moves in its report alone, #449's given-or-family being new in 2.3. At 1.4.0 and 2.0.0, which had no kana license, the three movers are parity, while `あ゙い たろう` and `山田 ア゙イ` move as every native-script CJK name does there, under the existing `fix(#271/#272/#298)` rule. RECOMPUTE: parse `ア゙イ タロウ`, `ア゛イ タロウ`, `マイケル ア゙イ`, `あ゙い たろう`, `山田 ア゙イ` and `김゙민준` here and on each baseline from PyPI.
472+
- 2026-10-03 #596 — A VOICING MARK TAKES THE SCRIPT OF THE KANA BEFORE IT. The four kana voicing marks U+3099–309C are in the hiragana BLOCK, which is how the script table decides, but none belongs to either syllabary: a mark voices the kana it follows. NFC composes one only where a precomposed voiced kana exists (カ + U+3099 is ガ), so `ア゙イ`, `ン゙` and every spacing ゛/゜ kept a hiragana-block character inside a katakana word, the word took the kana license, and a wholly-katakana name read family-first — the gap the #594 entry above measured, present since 2.1.0. DECIDED by deleting, from the classification copy only, every mark standing after a KANA, so the word reads as its base's script; token text is untouched. A mark after anything else — hangul, Han, Latin, or nothing — voices no kana and keeps the table's answer, as before. That scope is a review finding, not a nicety: the first cut deleted every mark with ANY character before it, which handed `김゙민준` the hangul script, and the surname division, which cuts the raw text, then split the mark off its base — given 김, family `゙민준`, unreported, where 2.3.0 left the word whole and reported given-or-family; `parser_for(locales.ZH)` cut `王゙小明` the same way. A case row watches `김゙민준` — marked tolerated, the reading of a mark after a non-kana being unpromised (rules.md's W Background) — so the radar tier, not the contract, sees it move. Declined: taking the marks out of the hiragana range instead. Measured 2026-10-03 on patched copies of this tree, it fixes the katakana names as well but leaves a word holding a mark in NO script, so `あ゙い たろう` loses family-first, `山田 ア゙イ` loses the kana license and reads family `ア゙イ`, and the unspaced `山田ア゙イ` reads given; the two parity case rows #596 added pin those readings. Declined too: narrowing to the two combining marks, which leaves `ア゛イ タロウ` family-first. NFD-typed text was the issue's worry and is not affected by either option, since classification composes first: NFD `やまだ がくと`, `ジョン ガイ` and `김 민준` parse identically under all of them. A mark with nothing before it keeps the table's answer, so `゙アイ タロウ` still reads family-first; that is degenerate input, recorded in rules.md's W Background as unpromised rather than given a branch. ONE ARGUMENT IN THE #594 ENTRY ABOVE NO LONGER HOLDS: an NFKC fold would no longer move a wholly-katakana name across W4's line, since the combining mark it produces (`ア゙` to `ア゙`) now reads as the katakana it follows — #594's own RECOMPUTE, run 2026-10-03 on this tree over `ア゙イ タロウ`, `ン゙ タロウ`, `山田 ア゙イ`, `マイケル ア゙イ`, `ア゚イ タロウ`, `山田 タロウ` and `ヤマダ タロウ` under all three orders, finds no role differing. The range decision stands on its other ground, the fold's reach past the kana (`JOHN`, `㈱`). The interpunct clause of rules.md#W4 was corrected in the same change: a name the 间隔号 divides keeps the DECLARED order, not its source order — under `name_order=FAMILY_FIRST` `威廉·莎士比亚` reads family 威廉 — and an example pins it. MEASURED 2026-10-03: no corpus line held an uncomposable mark before this change, so the case rows #596 added are the whole population. At 2.1.0, 2.2.0 and 2.3.0 the three `fix(#596)` rows move; at 2.0.0, 2.1.0 and 2.2.0 `김゙민준` moves in its report alone, #449's given-or-family being new in 2.3, and is parity at 1.4.0, which has no report. At 1.4.0 and 2.0.0, which had no kana license, the three movers are parity, while `あ゙い たろう` and `山田 ア゙イ` move as every native-script CJK name does there, under the existing `fix(#271/#272/#298)` rule. RECOMPUTE: parse `ア゙イ タロウ`, `ア゛イ タロウ`, `マイケル ア゙イ`, `あ゙い たろう`, `山田 ア゙イ` and `김゙민준` here and on each baseline from PyPI.
473473

474474
### D1 — the segmenterless-activation warning
475475

‎nameparser/_policy.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,8 @@ class Script(StrEnum):
140140
# combining U+3099, which is in the HIRAGANA block -- so the folded
141141
# token took the kana license and a wholly-katakana name turned
142142
# family-first (decisions.md#W4; #596 has since given every voicing
143-
# mark its base's script, see below). The span takes in the
144-
# halfwidth nakaguro U+FF65, which tokenize turns into a separator,
143+
# mark after a kana its base's script, see below). The span takes
144+
# in the halfwidth nakaguro U+FF65, which tokenize turns into a separator,
145145
# on the same direct-call grounds as U+30FB below, and the voicing
146146
# marks U+FF9E/U+FF9F: they are spacing characters (category Lm) that
147147
# NFC never composes into their base, there being no precomposed

‎tests/v2/cases.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9225,7 +9225,11 @@ def _check_cjk_shape_purity(self) -> None:
92259225
"of #596 dropped every mark with a character before it; "
92269226
"the token then classified as hangul, the surname site "
92279227
"divided the RAW text after 김, and the stranded mark "
9228-
"opened the family: given 김, family \u3099민준"),
9228+
"opened the family: given 김, family \u3099민준. "
9229+
"Tolerated: rules.md's W Background leaves a mark after "
9230+
"a non-kana unpromised, so this reading is watched on "
9231+
"the radar rather than pinned as contract",
9232+
tolerated=True),
92299233
Case("ja_iteration_mark_is_han", "佐々木 太郎",
92309234
{"family": "佐々木", "given": "太郎"},
92319235
classification="fix(#272)",

‎tests/v2/test_ledger_guards.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1151,6 +1151,8 @@ def test_case_shape_ids_exist_in_the_inventory() -> None:
11511151
("Anderson Smithさん", "Dr. Andersonさん", "Anderson, さん"),
11521152
"feat(#449) a glued Korean honorific leaves a Latin name word the script cannot order":
11531153
("Anderson Smith선생님", "Dr. Anderson선생님", "Anderson, 선생님"),
1154+
"feat(#449) a hangul word holding a stray voicing mark has no script, so the convention decides it":
1155+
("김민준", "김\u3099민준 지훈", "Dr. 김\u3099민준"),
11541156
# #491's two rules. Each boundary is a name the vocabulary reaches
11551157
# and the SHAPE does not: a title with an ordinary surname behind
11561158
# it, a credential with a name word beside it, and the lone title
@@ -3650,6 +3652,13 @@ def _claim(rule: dict) -> _Claim:
36503652
# move `title`, outside its fields, and go to the #594 rule;
36513653
# 'ヤマダ タロウ' is parity, as 'マイケル ジャクソン' already was
36523654
# inside the same class.
3655+
# 2026-10-03, #596: 147 -> 153, six new corpus names and not a
3656+
# wider rule: the five kana voicing-mark case rows and the
3657+
# tolerated hangul one ('김\u3099민준') lie in its script span.
3658+
# It explains 'あ\u3099い たろう' and '山田 ア\u3099イ', which move
3659+
# here as every native-script name does; the three fix(#596)
3660+
# movers are parity at this baseline, and the hangul row is
3661+
# parity too (v1 reads it first, as the tree does).
36533662
"fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots":
36543663
_Claim(153, ('family', 'given', 'middle'), "a94f0435fd18", None),
36553664
# 2026-09-19, #533: 33 -> 68. The count grew with the CORPUS
@@ -4699,6 +4708,8 @@ def _claim(rule: dict) -> _Claim:
46994708
"feat(#449) a glued Korean honorific leaves a Latin name word the script cannot order":
47004709
_Claim(1, ('_ambiguities', 'given', 'suffix'), "30ec95e25f05",
47014710
None),
4711+
"feat(#449) a hangul word holding a stray voicing mark has no script, so the convention decides it":
4712+
_Claim(1, ('_ambiguities',), "63f797a141db", None),
47024713
# #491's two rules, beside #449's for the same reason: both
47034714
# classify on `_ambiguities` alone, and no role moves anywhere
47044715
# in this change, so a widening that took a role would change
@@ -4808,6 +4819,13 @@ def _claim(rule: dict) -> _Claim:
48084819
# move `title`, outside its fields, and go to the #594 rule;
48094820
# 'ヤマダ タロウ' is parity, as 'マイケル ジャクソン' already was
48104821
# inside the same class.
4822+
# 2026-10-03, #596: 147 -> 153, six new corpus names and not a
4823+
# wider rule: the five kana voicing-mark case rows and the
4824+
# tolerated hangul one ('김\u3099민준') lie in its script span.
4825+
# It explains 'あ\u3099い たろう' and '山田 ア\u3099イ', which move
4826+
# here as every native-script name does; the three fix(#596)
4827+
# movers are parity at this baseline, and the hangul row is
4828+
# claimed first by its own #449 rule, on its report.
48114829
"fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots":
48124830
_Claim(153, ('_ambiguities', 'family', 'given', 'middle'), "a94f0435fd18", None),
48134831
# 37 -> 35 with the same 2026-09-05 narrowing as the 1.4 twin,

‎tools/differential/corpus_cjk.jsonl‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,6 @@
5757
"高橋一郎"
5858
"鵜殿"
5959
"김 민준"
60-
"김゙민준"
6160
"김민준"
6261
"김민준 님"
6362
"김민준 박사"

‎tools/differential/corpus_cjk_tolerated.jsonl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
"김, 민준씨"
3535
"김, 민준씨 (Jimmy)"
3636
"김. 민준"
37+
"김゙민준"
3738
"김민준 씨."
3839
"김민준 씨。"
3940
"김민준 씨."

‎tools/differential/expected_since_2.0.0.toml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ issue = "fix(#436/#437) a space-separated post-nominal run renders with spaces,
9191
name_regex = "^(?:JOHN DOE PHD MD|Jane Doe Jr\\. Ma|Jane Doe nee Puig i III|Jane Doe nee Smith PhD MA|John Doe MD PhD|John Smith MD PhD|John Smith Mc V|John Smith PhD Ed Ma|John Smith PhD Ma|Josep Lluis Carod i III|Josep Lluis Carod i V|Kenneth Clarke QC MP|Rovira, Josep Carod i Jr\\.|Smith, John PhD I\\.|The Rt Hon Kenneth Clarke QC MP, HMG|Washington Jr\\. MD, Franklin|abdul Smith Jr Ma|abdul Smith Jr V)$"
9292
fields = ["suffix"]
9393

94-
# The six #449 rules go SECOND, not first: the rule above
94+
# The seven #449 rules go SECOND, not first: the rule above
9595
# declares its own first-in-file position with a measurement,
9696
# and `fields = ["suffix"]` and `fields = ["_ambiguities"]`
9797
# are disjoint, so nothing orders the two against each other.
@@ -187,6 +187,17 @@ issue = "feat(#449) a glued Korean honorific leaves a Latin name word the script
187187
name_regex = "^Anderson선생님$"
188188
fields = ["_ambiguities", "given", "suffix"]
189189

190+
[[change]]
191+
issue = "feat(#449) a hangul word holding a stray voicing mark has no script, so the convention decides it"
192+
# '김゙민준' (김 + U+3099), a tolerated case row #596 added (2026-10-03).
193+
# A kana voicing mark after hangul voices no kana, so it keeps its
194+
# hiragana block and the token is mixed hangul and hiragana: no
195+
# script, no surname division, and O5's convention places the one name
196+
# word left. The roles are this baseline's; the report is the whole
197+
# diff. Literal-anchored, one corpus name, `_ambiguities` alone.
198+
name_regex = "^김\\u3099민준$"
199+
fields = ["_ambiguities"]
200+
190201
# The two #491 rules sit beside #449's, same position rule: both
191202
# declare `fields = ["_ambiguities"]`, so they are disjoint from the
192203
# `suffix` rule that owns first place and ahead of every rule below

0 commit comments

Comments
 (0)