convert: Fix handling of LLAMA2 vocab_size = -1

When vocab_size is detected to be -1 simply remove its value from the
parsed params.json and fallback to using the tok_embeddings.weight.

Fixes  #3900
This commit is contained in:
Nikolay Borisov 2023-11-06 13:03:31 +02:00
parent d6d905b242
commit f36a777bbc

View file

@ -253,6 +253,11 @@ class Params:
elif config["norm_eps"] in (1e-05, 1e-06):
# LLaMA v2
n_ctx = 4096
# For some reason FB writes -1 to vocab size for their LLAMA2 models
# simply remove this bogus value and let the return statement belo
# figure it out
if config["vocab_size"] == -1:
del config["vocab_size"]
else:
# LLaMA v1
n_ctx = 2048