vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

llama : add remove_space_prefix to llama_detokenize

Browse source 
This commit adds a new parameter to llama_detokenize to remove the
leading space before tokens if they have a word boundary character.

The motivation for this change is that when llama_server returns
completion_propabilities, the tokens are detokenized and currently
the leading space for the boundary tokens are removed. With this change
llama_server can set remove_space_prefix to false and the leading space
will be preserved.

Resolves: https://github.com/ggerganov/llama.cpp/issues/11728
This commit is contained in:
Daniel Bevenius 2025-02-10 09:47:18 +01:00
parent d7b31a9d84
commit cc1fd2fd0d
7 changed files with 35 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

4
include/llama.h
View file

@ -1025,6 +1025,7 @@ extern "C" {
/// @return Returns a negative number on failure - the number of chars/bytes that would have been returned.
/// @param remove_special Allow to remove BOS and EOS tokens if model is configured to do so.
/// @param unparse_special If true, special tokens are rendered in the output.
/// @param remove_space_prefix If true, removes the leading space before tokens if they have a word boundary character.
LLAMA_API int32_t llama_detokenize(
const struct llama_vocab * vocab,
const llama_token * tokens,
@ -1032,7 +1033,8 @@ extern "C" {
char * text,
int32_t text_len_max,
bool remove_special,
bool unparse_special);
bool unparse_special,
bool remove_space_prefix);
//
// Chat templates