vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add SPM infill support (#8016)

Browse source 
* add --spm-infill option

* support --spm-infill

* support --spm-infill
This commit is contained in:
Sigbjørn Skjæret 2024-06-28 12:53:43 +02:00 • committed by GitHub
parent b851b3fba0
commit 38373cfbab
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 34 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

1
examples/server/README.md
View file

@ -73,6 +73,7 @@ The project is under active development, and we are [looking for feedback and co
- `-fa`, `--flash-attn` : enable flash attention (default: disabled).
- `-ctk TYPE`, `--cache-type-k TYPE` : KV cache data type for K (default: `f16`, options `f32`, `f16`, `q8_0`, `q4_0`, `q4_1`, `iq4_nl`, `q5_0`, or `q5_1`)
- `-ctv TYPE`, `--cache-type-v TYPE` : KV cache type for V (default `f16`, see `-ctk` for options)
- `--spm-infill` : Use Suffix/Prefix/Middle pattern for infill (instead of Prefix/Suffix/Middle) as some models prefer this.
**If compiled with `LLAMA_SERVER_SSL=ON`**
- `--ssl-key-file FNAME`: path to file a PEM-encoded SSL private key