vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update README.md

Browse source
This commit is contained in:
John 2023-06-17 16:51:34 +02:00 • committed by JohannesGaessler
parent f9118b0ca5
commit 2797754843
1 changed files with 1 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
README.md
View file

@ -3,8 +3,7 @@ llama.cpp modification to run Falcon (work in progress)
Status: Status:
* Quantization works except for Q_K_ types * Quantization works except for Q_K_ types
* CUDA not yet functional * CUDA not yet functional
* context size calculation not proper (cuda as well as cpu) * python conversion script is very basic (produces ggml v0)
It appears the Q5 Falcon 40B inference time on CPU is as fast as the A100 fp16 inference time at 2 tk/second It appears the Q5 Falcon 40B inference time on CPU is as fast as the A100 fp16 inference time at 2 tk/second
CPU inference examples: CPU inference examples: