Skip to content

Upgrade lindera dependency from 2.0.1 to 5.0.1 - #36

Merged
mosuka merged 1 commit into
mainfrom
chore/upgrade-lindera-5
Aug 6, 2026
Merged

Upgrade lindera dependency from 2.0.1 to 5.0.1#36
mosuka merged 1 commit into
mainfrom
chore/upgrade-lindera-5

Conversation

@mosuka

@mosuka mosuka commented Aug 6, 2026

Copy link
Copy Markdown
Member

Summary

  • Move from lindera = "2.0.1" to lindera-analysis = "5.0.1". lindera 5.0.0 split the tokenizer/analysis-chain API (character filters, token filters, TokenizerBuilder/Tokenizer) out of the lindera crate into a new lindera-analysis crate; the pure segmenter stays in lindera. The API surface (TokenizerBuilder::new()/.build(), Tokenizer::tokenize(), Token.surface/.byte_start/.byte_end), the LINDERA_CONFIG_PATH env var, and the YAML config schema are all unchanged, so src/lib.rs/src/common.rs only needed an import-path swap — no tokenization logic changed.
  • Rework embed-* Cargo features to forward through lindera-analysis, and add embed-jieba.
  • Fix CI's test step (--all-features--features=embed-cjk) and a release.yml config-path typo; document which dictionary each embed-* feature embeds in the README.

Breaking changes

  1. embed-cjk now embeds Jieba instead of CC-CEDICT for Chinese. This mirrors upstream lindera 5.0.1's own embed-cjk definition (IPADIC + ko-dic + Jieba, changed from IPADIC + ko-dic + CC-CEDICT in lindera 2.x). Since lindera 5.x gates DictionaryKind::CcCedict behind the embed-cc-cedict feature specifically, anyone with dictionary: "embedded://cc-cedict" in their lindera.yml who builds with embed-cjk needs to either switch to the standalone embed-cc-cedict feature build, or repoint their config at embedded://jieba.
  2. Built artifacts are significantly larger. lindera 5.0.0 dropped the compress feature entirely (dictionaries are now embedded uncompressed for rkyv zero-copy loading). Locally, the embed-cjk release binary grew from 41.9 MB to 234 MB (~5.6x).

Why --all-features--features=embed-cjk in CI

--all-features would now build all 6 dictionaries uncompressed in a single CI job. Upstream lindera itself hit disk-space failures embedding even a single uncompressed dictionary per job (lindera#752) and had to add cleanup steps. Per-dictionary build coverage is already provided by this workflow's existing build job matrix (one job per embed-* feature), so switching the test job to embed-cjk only doesn't reduce coverage, just avoids stacking all 6 dictionaries in one job across 6 platforms including macOS/Windows.

Test plan

Closes #35

lindera 5.0.0 split the tokenizer/analysis-chain API (character filters,
token filters, TokenizerBuilder/Tokenizer) out of the `lindera` crate into
a new `lindera-analysis` crate; the pure segmenter stays in `lindera`.
Move this crate's only two usages (src/lib.rs, src/common.rs) to
`lindera_analysis::tokenizer`, which keeps the same API surface, the
`LINDERA_CONFIG_PATH` env var, and the YAML config schema.

Rework the embed-* Cargo features to forward through `lindera-analysis`
and add `embed-jieba`. `embed-cjk` is composed locally to match upstream
5.0.1's definition (IPADIC + ko-dic + Jieba); this is a breaking change
from the previous CC-CEDICT-based composition, since lindera 5.x gates
DictionaryKind::CcCedict behind the embed-cc-cedict feature specifically.

Switch CI's test step from --all-features to --features=embed-cjk: lindera
5 dropped dictionary compression, so --all-features would now build all 6
dictionaries uncompressed in a single job, and upstream lindera itself hit
disk-space failures embedding even one uncompressed dictionary per job.
Also fix release.yml's LINDERA_CONFIG_PATH pointing at a nonexistent
lindera.json, and document the dictionary embedded by each feature in the
README.
@mosuka
mosuka merged commit 3dc28b1 into main Aug 6, 2026
7 checks passed
@mosuka
mosuka deleted the chore/upgrade-lindera-5 branch August 6, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade lindera dependency from 2.0.1 to 5.0.1

1 participant