Upgrade lindera dependency from 2.0.1 to 5.0.1 - #36
Merged
Conversation
lindera 5.0.0 split the tokenizer/analysis-chain API (character filters, token filters, TokenizerBuilder/Tokenizer) out of the `lindera` crate into a new `lindera-analysis` crate; the pure segmenter stays in `lindera`. Move this crate's only two usages (src/lib.rs, src/common.rs) to `lindera_analysis::tokenizer`, which keeps the same API surface, the `LINDERA_CONFIG_PATH` env var, and the YAML config schema. Rework the embed-* Cargo features to forward through `lindera-analysis` and add `embed-jieba`. `embed-cjk` is composed locally to match upstream 5.0.1's definition (IPADIC + ko-dic + Jieba); this is a breaking change from the previous CC-CEDICT-based composition, since lindera 5.x gates DictionaryKind::CcCedict behind the embed-cc-cedict feature specifically. Switch CI's test step from --all-features to --features=embed-cjk: lindera 5 dropped dictionary compression, so --all-features would now build all 6 dictionaries uncompressed in a single job, and upstream lindera itself hit disk-space failures embedding even one uncompressed dictionary per job. Also fix release.yml's LINDERA_CONFIG_PATH pointing at a nonexistent lindera.json, and document the dictionary embedded by each feature in the README.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lindera = "2.0.1"tolindera-analysis = "5.0.1". lindera 5.0.0 split the tokenizer/analysis-chain API (character filters, token filters,TokenizerBuilder/Tokenizer) out of thelinderacrate into a newlindera-analysiscrate; the pure segmenter stays inlindera. The API surface (TokenizerBuilder::new()/.build(),Tokenizer::tokenize(),Token.surface/.byte_start/.byte_end), theLINDERA_CONFIG_PATHenv var, and the YAML config schema are all unchanged, sosrc/lib.rs/src/common.rsonly needed an import-path swap — no tokenization logic changed.embed-*Cargo features to forward throughlindera-analysis, and addembed-jieba.--all-features→--features=embed-cjk) and arelease.ymlconfig-path typo; document which dictionary eachembed-*feature embeds in the README.Breaking changes
embed-cjknow embeds Jieba instead of CC-CEDICT for Chinese. This mirrors upstream lindera 5.0.1's ownembed-cjkdefinition (IPADIC + ko-dic + Jieba, changed fromIPADIC + ko-dic + CC-CEDICTin lindera 2.x). Since lindera 5.x gatesDictionaryKind::CcCedictbehind theembed-cc-cedictfeature specifically, anyone withdictionary: "embedded://cc-cedict"in theirlindera.ymlwho builds withembed-cjkneeds to either switch to the standaloneembed-cc-cedictfeature build, or repoint their config atembedded://jieba.compressfeature entirely (dictionaries are now embedded uncompressed for rkyv zero-copy loading). Locally, theembed-cjkrelease binary grew from 41.9 MB to 234 MB (~5.6x).Why
--all-features→--features=embed-cjkin CI--all-featureswould now build all 6 dictionaries uncompressed in a single CI job. Upstream lindera itself hit disk-space failures embedding even a single uncompressed dictionary per job (lindera#752) and had to add cleanup steps. Per-dictionary build coverage is already provided by this workflow's existingbuildjob matrix (one job perembed-*feature), so switching thetestjob toembed-cjkonly doesn't reduce coverage, just avoids stacking all 6 dictionaries in one job across 6 platforms including macOS/Windows.Test plan
cargo build --features=embed-cjkLINDERA_CONFIG_PATH=./resources/lindera.yml cargo test --features=embed-cjk— 2 tests + 1 doctest pass, includingit_emits_segments's exact byte-offset assertions (the main regression risk, sinceOffsetMapping::correct_offsetwas internally rewritten upstream between v2 and v5)embed-*feature builds standalone with no cross-dependencycargo fmt --all -- --checkclean;cargo clippy --features=embed-cjkhas one pre-existing warning insrc/extension.rs, untouched by this PRsqlite3session, created an FTS5 table, inserted Japanese text, confirmedMATCHreturns correct results. (Loading against unpatchedmaincurrently fails due to the unrelated, already-open Extension fails to load on SQLite >= 3.47:fts5_api.iVersionis 3, but init requires exactly 2 #34 bug — reproduced identically on bothmainand this branch to confirm this PR isn't the cause; verified tokenization end-to-end with Extension fails to load on SQLite >= 3.47:fts5_api.iVersionis 3, but init requires exactly 2 #34's fix applied temporarily, then reverted before committing here since fixing Extension fails to load on SQLite >= 3.47:fts5_api.iVersionis 3, but init requires exactly 2 #34 is out of scope for this PR.)Closes #35