vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: readme

Browse source
This commit is contained in:
Trần Đức Nam 2023-12-22 15:18:52 +07:00
parent 6fcdb07773
commit b00e2d90f9
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
awqpy/README.md
View file

@ -43,7 +43,7 @@ Example for llama model
# For llama7b and llama2 models
python convert.py models/llama-7b/ --awq-path awq_cache/llama-7b-w4-g128.pt --outfile models/llama_7b_fp16.gguf
# For mistral and mpt models
python convert-hf-to-gguf.py models/llama-7b/ --awq-path awq_cache/llama-7b-w4-g128.pt --outfile models/llama_7b_fp16.gguf
python convert-hf-to-gguf.py models/mpt-7b/ --awq-path awq_cache/llama-7b-w4-g128.pt --outfile models/mpt_7b_fp16.gguf
```
## Quantize
@ -57,7 +57,7 @@ python convert-hf-to-gguf.py models/llama-7b/ --awq-path awq_cache/llama-7b-w4-g
# For llama and llama2, and mistral models.
./build/bin/main -m models/llama_7b_q4_0.gguf -n 128 --prompt "Once upon a time"
# For mpt models.
./build/bin/main -m models/llama_7b_q4_0.gguf -awq -n 128 --prompt "Once upon a time"
./build/bin/main -m models/mpt_7b_q4_0.gguf -awq -n 128 --prompt "Once upon a time"
```
## Benchmark