vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

add instructions in readme

Browse source
This commit is contained in:
caitianchi 2024-05-23 21:41:11 +08:00
parent 0480d5faa2
commit ec1cea7182
1 changed files with 22 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

30
examples/minicpmv/README.md
View file

@ -1,17 +1,31 @@
### 所有命令在 llama.cpp 根目录执行，模型位于根目录上级目录处
### All command should be executed under the root path of llama.cpp repo. We assume the MiniCPM-V-2.5 model are put in its parent folder.
## Instructions
Download model files from huggingface to "MiniCPM-V-2_5" folder.
Clone code
```bash
git clone -b minicpm-v2.5 https://github.com/OpenBMB/llama.cpp.git
cd llama.cpp
```
Prepare the model
```bash
python ./examples/minicpmv/minicpmv-surgery.py -m ../MiniCPM-V-2_5
python ./examples/minicpmv/minicpmv-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2_5 --minicpmv-projector ../MiniCPM-V-2_5/minicpmv.projector --output-dir ../MiniCPM-V-2_5/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5
python ./convert.py ../MiniCPM-V-2_5/model --outtype f16 --vocab-type bpe
# quantize int4 version
./quantize ../MiniCPM-V-2_5/model/model-8B-F16.gguf ../MiniCPM-V-2_5/model/ggml-model-Q4_K_M.gguf Q4_K_M
```
Try to inference
```bash
make
make minicpmv-cli
python ./examples/minicpmv/minicpmv-surgery.py -m ../MiniCPM-V-2_5
python ./examples/minicpmv/minicpmv-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2_5 --minicpmv-projector ../MiniCPM-V-2_5/minicpmv.projector --output-dir ../MiniCPM-V-2_5/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5
python ./convert.py ../MiniCPM-V-2_5/model --outtype f16 --vocab-type bpe
# run quantize f16 version
./minicpmv-cli -m ../MiniCPM-V-2_5/model/model-8B-F16.gguf --mmproj ../MiniCPM-V-2_5/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?"
# or run quantize int4 version
./quantize ../MiniCPM-V-2_5/model/model-8B-F16.gguf ../MiniCPM-V-2_5/model/ggml-model-Q4_K_M.gguf Q4_K_M
# run quantize int4 version
./minicpmv-cli -m ../MiniCPM-V-2_5/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_5/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?"
# or run in interactive mode
@ -20,7 +34,7 @@ python ./convert.py ../MiniCPM-V-2_5/model --outtype f16 --vocab-type bpe
### Android
#### Build on Android using Termux
#### Build for Android using Termux
[Termux](https://github.com/termux/termux-app#installation) is a method to execute `llama.cpp` on an Android device (no root required).
```
apt update && apt upgrade -y