vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

server : support reading arguments from environment variables (#9105)

Browse source 
* server : support reading arguments from environment variables

* add -fa and -dt

* readme : specify non-arg env var
This commit is contained in:
Xuan Son Nguyen 2024-08-21 11:04:34 +02:00 • committed by GitHub
parent b40eb84895
commit fc54ef0d1c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 80 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

19
examples/server/README.md
View file

@ -247,6 +247,25 @@ logging:
--log-append Don't truncate the old log file.
```
Available environment variables (if specified, these variables will override parameters specified in arguments):
- `LLAMA_CACHE` (cache directory, used by `--hf-repo`)
- `HF_TOKEN` (Hugging Face access token, used when accessing a gated model with `--hf-repo`)
- `LLAMA_ARG_MODEL`
- `LLAMA_ARG_THREADS`
- `LLAMA_ARG_CTX_SIZE`
- `LLAMA_ARG_N_PARALLEL`
- `LLAMA_ARG_BATCH`
- `LLAMA_ARG_UBATCH`
- `LLAMA_ARG_N_GPU_LAYERS`
- `LLAMA_ARG_THREADS_HTTP`
- `LLAMA_ARG_CHAT_TEMPLATE`
- `LLAMA_ARG_N_PREDICT`
- `LLAMA_ARG_ENDPOINT_METRICS`
- `LLAMA_ARG_ENDPOINT_SLOTS`
- `LLAMA_ARG_EMBEDDINGS`
- `LLAMA_ARG_FLASH_ATTN`
- `LLAMA_ARG_DEFRAG_THOLD`
## Build

3
examples/server/server.cpp
View file

@ -2507,6 +2507,9 @@ int main(int argc, char ** argv) {
return 1;
}
// parse arguments from environment variables
gpt_params_parse_from_env(params);
// TODO: not great to use extern vars
server_log_json = params.log_json;
server_verbose = params.verbosity > 0;