vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

convert : use presence of tokenizer.json to determine StableLM tokenizer loader

Browse source 
It's a less arbitrary heuristic than the vocab size.
This commit is contained in:
Francis Couture-Harpin 2024-01-21 11:01:31 -05:00
parent a11f1497ef
commit b7b53a5ccc
1 changed files with 4 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

8
convert-hf-to-gguf.py
View file

@ -929,11 +929,11 @@ class PersimmonModel(Model):
class StableLMModel(Model):
def set_vocab(self):
# StableLM 2 1.6B uses a vocab in a similar format to Qwen's vocab
if self.hparams["vocab_size"] > 100000:
self._set_vocab_qwen()
else:
if (self.dir_model / "tokenizer.json").is_file():
self._set_vocab_gpt2()
else:
# StableLM 2 1.6B uses a vocab in a similar format to Qwen's vocab
self._set_vocab_qwen()
def set_gguf_parameters(self):
hparams = self.hparams