vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update README.md

Browse source
This commit is contained in:
John 2023-06-17 16:23:01 +02:00 • committed by JohannesGaessler
parent d0c460629d
commit 9d4d26554a
1 changed files with 4 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

6
README.md
View file

@ -1,10 +1,12 @@
llama.cpp modification to run Falcon (work in progress) llama.cpp modification to run Falcon (work in progress)
Status: Status:
Quantization works except for Q_K_ types * Quantization works except for Q_K_ types
CUDA not yet functional * CUDA not yet functional
*
It appears the Q5 Falcon 40B inference time on CPU is as fast as the A100 fp16 inference time at 2 tk/second It appears the Q5 Falcon 40B inference time on CPU is as fast as the A100 fp16 inference time at 2 tk/second
CPU inference examples:
``` ```
Q:\ggllm.cpp> .\build\bin\Release\falcon_main.exe -t 31 -m Q:\models\falcon-40b\q5_1 -p "Love relates to hate like" -n 50 -ngl 0 Q:\ggllm.cpp> .\build\bin\Release\falcon_main.exe -t 31 -m Q:\models\falcon-40b\q5_1 -p "Love relates to hate like" -n 50 -ngl 0
main: build = 677 (dd3d346) main: build = 677 (dd3d346)