vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update LICENSE and TODOs in README (#14)

Browse source 
* Update LICENSE with our copyright notice

* Update README.md

* fix readme anchor

* Update README.md
This commit is contained in:
Holden X 2023-12-19 16:23:10 +08:00 • committed by GitHub
parent 7cebaba8e4
commit e3b4b85caa
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 12 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

1
LICENSE
View file

@ -1,6 +1,7 @@
MIT License MIT License
Copyright (c) 2023 Georgi Gerganov Copyright (c) 2023 Georgi Gerganov
Copyright (c) 2023 SJTU-IPADS
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal

21
README.md
View file

@ -53,14 +53,19 @@ We have tested PowerInfer on the following platforms:
- x86-64 CPU (with AVX2 instructions) on Linux - x86-64 CPU (with AVX2 instructions) on Linux
- x86-64 CPU and NVIDIA GPU on Linux - x86-64 CPU and NVIDIA GPU on Linux
- Apple M Chips on macOS (As we do not optimize for Mac, the performance improvement is not significant now.) - Apple M Chips on macOS (As we do not optimize for Mac, the performance improvement is not significant now.)
And new features coming soon:
- Mistral-7B model
- Online fine-grained FFN offloading to GPU
- Metal backend for sparse inference on macOS
## Getting Started ## Getting Started
- [Installation](##setup--installation) - [Installation](#setup-and-installation)
- [Model Weights](##model-weights) - [Model Weights](#model-weights)
## Setup & Installation ## Setup and Installation
### Get the Code ### Get the Code
```bash ```bash
@ -70,12 +75,7 @@ cd PowerInfer
### Build ### Build
In order to build PowerInfer you have two different options. These commands are supposed to be run from the root directory of the project. In order to build PowerInfer you have two different options. These commands are supposed to be run from the root directory of the project.
Using `make` on Linux or macOS: Using `CMake` on Linux or macOS:
```bash
make
```
Using `CMake`:
* If you have one GPU: * If you have one GPU:
```bash ```bash
cmake -S . -B build -DLLAMA_CUBLAS=ON cmake -S . -B build -DLLAMA_CUBLAS=ON
@ -130,6 +130,7 @@ PowerInfer achieves up to 11x and 8x speedup for FP16 and INT4 models!
We will release the code and data in the following order, please stay tuned! We will release the code and data in the following order, please stay tuned!
- [x] Release core code of PowerInfer, supporting Llama-2, Falcon-40B. - [x] Release core code of PowerInfer, supporting Llama-2, Falcon-40B.
- [ ] Support Mistral-7B
- [ ] Release perplexity evaluation code - [ ] Release perplexity evaluation code
- [ ] Support Metal for Mac - [ ] Support Metal for Mac
- [ ] Release code for OPT models - [ ] Release code for OPT models