Skip to content

chore(release): ship a matching lindera.yml in every release archive - #38

Merged
mosuka merged 1 commit into
lindera:mainfrom
Saeris:chore/ship-config-with-release-archives
Aug 8, 2026
Merged

chore(release): ship a matching lindera.yml in every release archive#38
mosuka merged 1 commit into
lindera:mainfrom
Saeris:chore/ship-config-with-release-archives

Conversation

@Saeris

@Saeris Saeris commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up to the FTS5 fix (#37), split out so it can be reviewed on its own. It addresses a secondary issue mentioned in #34. I'll leave it as a draft until the base FTS5 fix branch is merged, after which I'll rebase this on main in case any changes get added during review.

The problem

The release archives contain only the shared library. LINDERA_CONFIG_PATH is required for the tokenizer to load, so a user of a prebuilt extension has to discover that requirement, then go find a configuration somewhere undocumented.

Also, resources/lindera.yml is also IPADIC-specific, it names embedded://ipadic and filters on IPADIC part-of-speech tags. Five of the six release variants embed a different dictionary, so copying it across is the wrong configuration for all of them.

The fix

This adds a configuration per embedded dictionary and gives each build matrix entry a config field naming its own, so every archive carries the configuration matching the dictionary it actually embedded.

It is staged as a plain lindera.yml inside the archive: the variant is already in the archive's filename, and a fixed name keeps the setup instructions identical for every download.

Feature Configuration
embed-ipadic, embed-cjk resources/lindera.yml (existing)
embed-unidic resources/lindera-unidic.yml
embed-ko-dic resources/lindera-ko-dic.yml
embed-cc-cedict resources/lindera-cc-cedict.yml
embed-jieba resources/lindera-jieba.yml

embed-cjk embeds three dictionaries but a configuration selects one, so it keeps the existing IPADIC default.

How the new configurations were derived

Each mirrors resources/lindera.yml, adjusted only where the dictionary requires it. Every filter name and tag value was checked against lindera-analysis-5.0.1 rather than inferred:

  • UniDic: japanese_stop_tags is restricted to the top-level tags IPADIC and UniDic agree on (助詞, 助動詞, 記号). The IPADIC config's deeper tags (助詞,格助詞,一般) subdivide differently in UniDic, so carrying them over would silently match nothing.
  • ko-dic: korean_stop_tags with EP/EF/JKG, the Korean counterparts of the 助動詞/助詞 the IPADIC config drops. japanese_iteration_mark is dropped as inapplicable.
  • CC-CEDICT, Jieba: segmentation plus unicode_normalize only. Lindera ships no Chinese-specific filters, and the Japanese and Korean stop-tag filters key on tag vocabularies these dictionaries do not use. The language-neutral filters (stop_words, length, lowercase) remain available to anyone who wants them.

remove_diacritical_mark keeps japanese: false in the Japanese configs, as in the existing one. Stripping dakuten would collapse か/が and は/ば/ぱ into a single token.

Also

The README documented only the build-from-source path. Added a short prebuilt-extension section and a feature→configuration table.

Verification

  • Confirmed all five embedded:// dictionary URIs (ipadic, unidic, ko-dic, cc-cedict, jieba) against the lindera 5.0.1 sources.
  • Confirmed every filter name used (unicode_normalize, japanese_iteration_mark, japanese_stop_tags, japanese_katakana_stem, korean_stop_tags, remove_diacritical_mark) against the *_FILTER_NAME constants in lindera-analysis-5.0.1, and the EP/EF/JKG values against that crate's own ko-dic fixtures.
  • Ran the Windows archive step locally: Compress-Archive -Path <dll>,lindera.yml places both entries flat at the archive root, matching zip --junk-paths, so the layout is identical across all three platforms.

The release archives contain only the shared library, so anyone using a
prebuilt extension has to know that LINDERA_CONFIG_PATH is required, then
find a configuration for it in a source tree they never cloned.

resources/lindera.yml is also IPADIC-specific: it names embedded://ipadic
and filters on IPADIC part-of-speech tags. Five of the six release variants
embed a different dictionary, so it is the wrong configuration for them.

Add a configuration per embedded dictionary and give each build matrix entry
a `config` field naming its own, so every archive carries the one matching
the dictionary it embedded. It is staged as a plain `lindera.yml` inside the
archive, since the variant is already in the archive's filename and a fixed
name keeps the setup identical everywhere.

The new configurations mirror resources/lindera.yml, adjusted per dictionary:

- unidic: japanese_stop_tags is restricted to the top-level tags IPADIC and
  UniDic agree on. The IPADIC config's deeper tags (助詞,格助詞,一般) subdivide
  differently in UniDic and would silently match nothing.
- ko-dic: korean_stop_tags with EP/EF/JKG, the Korean counterparts of the
  助動詞/助詞 the IPADIC config drops. japanese_iteration_mark is dropped.
- cc-cedict, jieba: segmentation plus unicode_normalize only. Lindera ships
  no Chinese-specific filters, and the Japanese and Korean stop-tag filters
  key on tag vocabularies these dictionaries do not use.

embed-cjk keeps resources/lindera.yml: it embeds three dictionaries but a
configuration selects one, and IPADIC is the existing default.

Also document the prebuilt path in the README, which until now covered only
building from source.
@Saeris
Saeris force-pushed the chore/ship-config-with-release-archives branch from cea5d08 to c54600e Compare August 7, 2026 20:04
@Saeris
Saeris marked this pull request as ready for review August 7, 2026 20:05

@mosuka mosuka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@mosuka
mosuka merged commit 6aed060 into lindera:main Aug 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants