vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add ability to use Q5_0, Q5_1, and IQ4_NL for quantized K cache (#6183)

Browse source 
* k_cache: be able to use Q5_0

* k_cache: be able to use Q5_1 on CODA

* k_cache: be able to use Q5_0 on Metal

* k_cache: be able to use Q5_1 on Metal

* k_cache: be able to use IQ4_NL - just CUDA for now

* k_cache: be able to use IQ4_NL on Metal

* k_cache: add newly added supported types to llama-bench and CUDA supports_op

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
This commit is contained in:
Kawrakow 2024-03-21 08:27:57 +01:00 • committed by GitHub
parent c5b8595e3f
commit 76aa30a263
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 424 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

3
common/common.cpp
View file

@ -1590,6 +1590,9 @@ static ggml_type kv_cache_type_from_str(const std::string & s) {
if (s == "q4_1") {
return GGML_TYPE_Q4_1;
}
if (s == "iq4_nl") {
return GGML_TYPE_IQ4_NL;
}
if (s == "q5_0") {
return GGML_TYPE_Q5_0;
}