vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add comment explaining the logic behind the if statement

Browse source
This commit is contained in:
Billel Mokeddem 2024-12-18 05:46:07 +00:00
parent fc055407b7
commit b3d022aa1a
1 changed files with 1 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
convert_hf_to_gguf.py
View file

@ -527,6 +527,7 @@ class Model:
if token in added_vocab: if token in added_vocab:
# We need to manually encode and decode the added tokens in case special characters # We need to manually encode and decode the added tokens in case special characters
# used for `\n` / `\t` have been manually added in the added tokens # used for `\n` / `\t` have been manually added in the added tokens
# To avoid unexpected issues - we make sure to encode single-char tokens
if len(token) == 1: if len(token) == 1:
token = tokenizer.decode(tokenizer.encode(token, add_special_tokens=False)) token = tokenizer.decode(tokenizer.encode(token, add_special_tokens=False))