Update the llama.cpp submodule to b10405 and port two API changes - #1097
Open
bogdanr wants to merge 1 commit into
Open
Update the llama.cpp submodule to b10405 and port two API changes#1097bogdanr wants to merge 1 commit into
bogdanr wants to merge 1 commit into
Conversation
The submodule moves from b10200 to b10405. Two functions in the sampler API change in this range. The Rust wrappers now agree with them. llama_sampler_init_penalties has a new first parameter, n_vocab. Upstream moves n_vocab from llama_sampler_data into the penalty sampler (#26520). LlamaSampler::penalties has the same new first parameter. This is a breaking change, and each caller must add n_vocab as the first argument. The mirostat and logit_bias wrappers already take n_vocab as the first parameter, so the penalties wrapper is now consistent with them. llama_sampler_init_dry no longer has the n_ctx_train parameter. Upstream removes the full-context window from the samplers that keep a history (#26524). LlamaSampler::dry keeps its public signature. The wrapper no longer reads n_ctx_train from the model. The same upstream change also changes the meaning of penalty_last_n. A negative value selected the context length. Upstream now sets a negative value to 0, and 0 disables the penalty. The documentation of LlamaSampler::penalties shows the new behaviour. The reason for this update is the output of the DeepSeek-V4 models. The current submodule gives incorrect text for these models. A build from b10405 gives correct text for the same model, the same prompt and the same hardware. cargo build --workspace, cargo clippy and cargo test --features sampler are clean with the new submodule. Signed-off-by: Bogdan Radulescu <bogdan@nimblex.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The submodule moves from b10200 to b10405. Two functions in the sampler API change in this range. The Rust wrappers now agree with them.
llama_sampler_init_penaltieshas a new first parameter,n_vocab. Upstream movesn_vocabfromllama_sampler_datainto the penalty sampler (ggml-org/llama.cpp#26520).LlamaSampler::penaltieshas the same new first parameter. This is a breaking change, and each caller must addn_vocabas the first argument. Themirostatandlogit_biaswrappers already taken_vocabas the first parameter, so thepenaltieswrapper is now consistent with them.llama_sampler_init_dryno longer has then_ctx_trainparameter. Upstream removes the full-context window from the samplers that keep a history (ggml-org/llama.cpp#26524).LlamaSampler::drykeeps its public signature. The wrapper no longer readsn_ctx_trainfrom the model.The same upstream change also changes the meaning of
penalty_last_n. A negative value selected the context length. Upstream now sets a negative value to 0, and 0 disables the penalty. The documentation ofLlamaSampler::penaltiesshows the new behaviour.The reason for this update is the output of the DeepSeek-V4 models. The current submodule gives incorrect text for these models. A build from b10405 gives correct text for the same model, the same prompt and the same hardware.
cargo build --workspace,cargo clippyandcargo test --features samplerare clean with the new submodule.One note on CI:
cargo fmt --checkalready fails onmaininexamples/llguidance.rs,llama-cpp-2/src/llguidance_sampler.rsandllama-cpp-2/src/model.rs. This branch does not add to that list, and it does not correct it, because the files are not related to this change. Tell me if you prefer a separate pull request for the format.