vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

llama : add remove_space_prefix to llama_detokenize

Browse source 
This commit adds a new parameter to llama_detokenize to remove the
leading space before tokens if they have a word boundary character.

The motivation for this change is that when llama_server returns
completion_propabilities, the tokens are detokenized and currently
the leading space for the boundary tokens are removed. With this change
llama_server can set remove_space_prefix to false and the leading space
will be preserved.

Resolves: https://github.com/ggerganov/llama.cpp/issues/11728
This commit is contained in:
Daniel Bevenius 2025-02-10 09:47:18 +01:00
parent d7b31a9d84
commit cc1fd2fd0d
7 changed files with 35 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

4
common/llguidance.cpp
View file

@ -176,12 +176,12 @@ static LlgTokenizer * llama_sampler_llg_new_tokenizer(const llama_vocab * vocab)
llama_token token = i;
auto dp = (char *) token_bytes + offset;
auto size = llama_detokenize(vocab, &token, 1, dp, max_token, false, false);
auto size = llama_detokenize(vocab, &token, 1, dp, max_token, false, false, true);
if (size < 0) {
GGML_ABORT("llama_detokenize failed\n");
}
if (size == 0) {
size = llama_detokenize(vocab, &token, 1, dp + 1, max_token - 1, false, true);
size = llama_detokenize(vocab, &token, 1, dp + 1, max_token - 1, false, true, true);
if (size < 0) {
GGML_ABORT("llama_detokenize failed\n");
}