From ded0613bd43da937cf80d3f52b2b27a1b0d98cc6 Mon Sep 17 00:00:00 2001 From: Holden X Date: Tue, 19 Dec 2023 17:09:33 +0800 Subject: [PATCH] Fix issues in README on feedback (#15) * Update README.md * Update README.md * Update README.md * Update README.md --------- Co-authored-by: Jeremy Song <76689794+YixinSong-e@users.noreply.github.com> --- README.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 5e83cc722..f2a4355d7 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ PowerInfer is a CPU/GPU LLM inference engine leveraging **activation locality** ## Demo 🔥 -https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/d26ae05b-d0cf-40b6-8788-bda3fe447e28 +https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/fe441a42-5fce-448b-a3e5-ea4abb43ba23 PowerInfer v.s. llama.cpp on a single RTX 4090(24G) running Falcon(ReLU)-40B-FP16 with a 11x speedup! @@ -75,8 +75,8 @@ cd PowerInfer ### Build In order to build PowerInfer you have two different options. These commands are supposed to be run from the root directory of the project. -Using `CMake` on Linux or macOS: -* If you have one GPU: +Using `CMake`(3.13+) on Linux or macOS: +* If you have an NVIDIA GPU: ```bash cmake -S . -B build -DLLAMA_CUBLAS=ON cmake --build build --config Release @@ -109,14 +109,7 @@ If you want to limit the VRAM usage of GPU: ./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt --vram-budget $vram_gb ``` -As for now, it requires an offline-generated "GPU index" file to split FFNs on GPU. If you want to try it, please use the following instructions to generate the GPU index file: -```bash -python scripts/export-gpu-split.py $activation_count_path $output_idx_path solver -``` -Then, you can use the following instructions to run PowerInfer with GPU index: -```bash -./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt --gpu-index $split_path -``` +As for now, it requires an offline-generated "GPU index" file to split FFNs on GPU. And we found these files are hard to maintain and distribute. We will ship automatic FFN split based on VRAM capacity via [#11](https://github.com/SJTU-IPADS/PowerInfer/pull/11) very soon. ## Evaluation @@ -131,6 +124,8 @@ We will release the code and data in the following order, please stay tuned! - [x] Release core code of PowerInfer, supporting Llama-2, Falcon-40B. - [ ] Support Mistral-7B +- [ ] Support Windows +- [ ] Support text-generation-webui - [ ] Release perplexity evaluation code - [ ] Support Metal for Mac - [ ] Release code for OPT models