vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add benchmarks

Browse source
This commit is contained in:
Yuval Peled 2023-06-02 15:27:08 +03:00
parent 83cd67dda6
commit 9291f4c606
1 changed files with 9 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/token_generation_performance_tips.md
View file

@ -25,10 +25,16 @@ GPU: A6000 (48GB VRAM)
CPU: 7 physical cores CPU: 7 physical cores
RAM: 32GB RAM: 32GB
Model: `TheBloke_Wizard-Vicuna-30B-Uncensored-GGML/Wizard-Vicuna-30B-Uncensored.ggmlv3.q4_0.bin` (30B parameters, 4bit quantization, GGML)
Run command: `./main -m "path/to/model.bin" -p "-p "An extremely detailed description of the 10 best ethnic dishes will follow, with recipes: " -n 1000 [additional benchmark flags]`
Result: Result:
| command | tokens/second | | command | tokens/second (higher is better) |
| - | - | | - | - |
| -ngl 2000000 | N/A (less than 0.1) | | -ngl 2000000 | N/A (less than 0.1) |
| -t 7 | 1.6 | | -t 7 | 1.7 |
| -t 7 -ngl 2000000 | 8.5 | | -t 1 -ngl 2000000 | 5.5 |
| -t 7 -ngl 2000000 | 8.7 |
| -t 4 -ngl 2000000 | 9.1 |