vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add documents about Vulkan

Browse source
This commit is contained in:
calvinweb 2024-01-29 20:09:36 +08:00 • committed by GitHub
parent e76627bcce
commit b51e79422c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 7 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
README.md
View file

@ -344,7 +344,14 @@ Finally, you're ready to run a computation using `mpirun`:
```bash
mpirun -hostfile hostfile -n 3 ./main -m ./models/7B/ggml-model-q4_0.gguf -n 128
```
### Vulkan Build
**Vulkan** is a new library aimed at replacing **OpenGL**. Starting from version `b1996`, `llama.cpp` now supports **Vulkan** as a backend. It can be faster (almost 2x on NVIDIA GeForce RTX™ 3060 Laptop GPU) than **CUDA** on a consumer-grade NVIDIA graphics card.
To build with **Vulkan**:
- Ensure that your operating system and graphics card support **Vulkan**.
- Build `llama.cpp` with the following arguments: `LLAMA_VULKAN=1`.
### BLAS Build
Building the program with BLAS support may lead to some performance improvements in prompt processing using batch sizes higher than 32 (the default is 512). Support with CPU-only BLAS implementations doesn't affect the normal generation performance. We may see generation performance improvements with GPU-involved BLAS implementations, e.g. cuBLAS, hipBLAS and CLBlast. There are currently several different BLAS implementations available for build and use: