convert: Fix handling of LLAMA2 vocab_size = -1
When vocab_size is detected to be -1 simply remove its value from the parsed params.json and fallback to using the tok_embeddings.weight. Fixes #3900
This commit is contained in:
parent
d6d905b242
commit
f36a777bbc
1 changed files with 5 additions and 0 deletions
|
@ -253,6 +253,11 @@ class Params:
|
|||
elif config["norm_eps"] in (1e-05, 1e-06):
|
||||
# LLaMA v2
|
||||
n_ctx = 4096
|
||||
# For some reason FB writes -1 to vocab size for their LLAMA2 models
|
||||
# simply remove this bogus value and let the return statement belo
|
||||
# figure it out
|
||||
if config["vocab_size"] == -1:
|
||||
del config["vocab_size"]
|
||||
else:
|
||||
# LLaMA v1
|
||||
n_ctx = 2048
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue