Add AVX2 support for x86 architectures thanks to @Const-me !
This commit is contained in:
parent
a9e58529ea
commit
f1eaff4721
2 changed files with 166 additions and 3 deletions
|
@ -17,6 +17,7 @@ The main goal is to run the model using 4-bit quantization on a MacBook.
|
|||
|
||||
- Plain C/C++ implementation without dependencies
|
||||
- Apple silicon first-class citizen - optimized via Arm Neon and Accelerate framework
|
||||
- AVX2 support for x86 architectures
|
||||
- Mixed F16 / F32 precision
|
||||
- 4-bit quantization support
|
||||
- Runs on the CPU
|
||||
|
@ -185,9 +186,6 @@ When running the larger models, make sure you have enough disk space to store al
|
|||
In general, it seems to work, but I think it fails for unicode character support. Hopefully, someone can help with that
|
||||
- I don't know yet how much the quantization affects the quality of the generated text
|
||||
- Probably the token sampling can be improved
|
||||
- x86 quantization support [not yet ready](https://github.com/ggerganov/ggml/pull/27). Basically, you want to run this
|
||||
on Apple Silicon. For now, on Linux and Windows you can use the F16 `ggml-model-f16.bin` model, but it will be much
|
||||
slower.
|
||||
- The Accelerate framework is actually currently unused since I found that for tensor shapes typical for the Decoder,
|
||||
there is no benefit compared to the ARM_NEON intrinsics implementation. Of course, it's possible that I simlpy don't
|
||||
know how to utilize it properly. But in any case, you can even disable it with `LLAMA_NO_ACCELERATE=1 make` and the
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue