Georgi Gerganov
546112944a
Merge branch 'master' into HEAD
2023-09-28 21:35:17 +03:00
Georgi Gerganov
2619109ad5
ci : disable freeBSD builds due to lack of VMs ( #3381 )
2023-09-28 19:36:36 +03:00
Georgi Gerganov
ec893798b7
llama : custom attention mask + parallel decoding + no context swaps ( #3228 )
...
* tests : verify that RoPE is "additive"
* llama : replace ggml_diag_mask_inf with ggml_add (custom -inf mask)
* ggml : ggml_rope now takes a vector with positions instead of n_past
* metal : add rope_f16 kernel + optimize cpy kernels
* llama : unified KV cache + batch inference API
* llama : add new llama_decode() API that works with llama_batch
* llama : add cell_max heuristic for more efficient kv_cache
* llama : extend llama_kv_cache API
* llama : more robust cell_max heuristic + wip shift
* metal : disable concurrency optimization
* llama : add llama_kv_cache_shift_seq + no more context swaps
* llama : apply K-cache roping for Falcon and Baichuan
* speculative : fix KV cache management
* parallel : example for serving multiple users in parallel
* parallel : disable hot-plug to avoid cache fragmentation
* fixes : speculative KV cache + llama worst-case graph
* llama : extend batch API to select which logits to output
* llama : fix worst case graph build
* ggml-cuda : update rope implementation for parallel decoding (#3254 )
* ggml-cuda : update rope implementation for parallel decoding
* better solution for p0 computation
* fix rope
* simpler rope implementation
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* make : add parallel to build + fix static functions in llama.cpp
* simple : fix token counting
* parallel : various improvements
* llama : fix cell_max logic + rename functions
* parallel : try smaller batches when the KV cache is fragmented
* parallel : fix sequence termination criteria
* llama : silence errors KV cache errors
* parallel : remove new line from prompt
* parallel : process system prompt once + configurable paramters + llama API
* parallel : remove question with short answers
* parallel : count cache misses
* parallel : print misses on each request
* parallel : minor
* llama : fix n_kv to never become 0
* parallel : rename hot-plug to continuous-batching
* llama : improve llama_batch API + simplify parallel example
* simple : add parallel decoding support
* simple : improve comments + free batch
* ggml-cuda : add rope f16, restore performance with parallel decoding (#3272 )
* ggml-cuda : add rope f16, restore performance
* offload KQ_mask with all models
* fix rope shift
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : disable MPI for now
ggml-ci
* train : make KQ_pos memory buffer permanent via dummy scale op
* ggml : revert change to ggml_cpy, add ggml_cont_Nd instead (#3275 )
ggml-ci
* parallel : fix bug (extra BOS) + smaller token_prev array
* parallel : fix cases where the input prompts can overflow the batch
* parallel : add disabled experimental batch chunking in powers of two
* llama : llama.h formatting + comments
* simple : add README.md
* llama : fix kv cache heuristic when context is less than 32
* parallel : fix crash when `-n -1`
* llama : simplify returns if/else branches
* metal : use mm kernels for batch size > 2
* examples : utilize new llama_get_logits_ith()
* examples : add example for batched decoding
* examples : do not eval prompt 2 times (close #3348 )
* server : clear the KV cache beyond n_past before llama_decode
* server : avoid context swaps by shifting the KV cache
---------
Co-authored-by: slaren <slarengh@gmail.com>
2023-09-28 19:04:36 +03:00
Kevin Ji
45855b3f1c
docs : mark code as Bash ( #3375 )
2023-09-28 09:11:32 -04:00
Pierre Alexandre SCHEMBRI
4aea3b846e
readme : add Mistral AI release 0.1 ( #3362 )
2023-09-28 15:13:37 +03:00
slaren
da0400344b
ggml-cuda : perform cublas fp16 matrix multiplication as fp16 ( #3370 )
...
Code Coverage / run (push) Has been cancelled
* ggml-cuda : perform cublas fp16 matrix multiplication as fp16
* try to fix rocm build
* restrict fp16 mat mul to volta and up
2023-09-28 13:08:28 +03:00
Zhang Peiyuan
e519621010
convert : remove bug in convert.py permute function ( #3364 )
2023-09-27 20:45:20 +02:00
Richard Roberson
ac43576124
make-ggml.py : compatibility with more models and GGUF ( #3290 )
...
* Resync my fork with new llama.cpp commits
* examples : rename to use dash instead of underscore
* New model conversions
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-09-27 19:25:12 +03:00
Cebtenzzre
20c7e1e804
gguf : fix a few general keys ( #3341 )
2023-09-27 12:18:07 -04:00
Rickard Hallerbäck
dc6897404e
metal : reusing llama.cpp logging ( #3152 )
...
* metal : reusing llama.cpp logging
* cmake : build fix
* metal : logging callback
* metal : logging va_args memory fix
* metal : minor cleanup
* metal : setting function like logging macro to capital letters
* llama.cpp : trailing whitespace fix
* ggml : log level enum used by llama
* Makefile : cleanup ggml-metal recipe
* ggml : ggml_log_callback typedef
* ggml : minor
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-09-27 18:48:33 +03:00
Jag Chadha
527e57cfd8
build : add ACCELERATE_NEW_LAPACK to fix warning on macOS Sonoma ( #3342 )
2023-09-27 18:34:32 +03:00
BarfingLemurs
ffe88a36a9
readme : add some recent perplexity and bpw measurements to READMES, link for k-quants ( #3340 )
...
* Update README.md
* Update README.md
* Update README.md with k-quants bpw measurements
2023-09-27 18:30:36 +03:00
DAN™
99115f3fa6
cmake : fix build-info.h on MSVC ( #3309 )
2023-09-25 18:45:33 -04:00
2f38b454
1726f9626f
docs: Fix typo CLBlast_DIR var. ( #3330 )
2023-09-25 20:24:52 +02:00
Erik Scholz
a98b1633d5
nix : add cuda, use a symlinked toolkit for cmake ( #3202 )
2023-09-25 13:48:30 +02:00
xaedes
166065837e
improve handling of not yet supported tensor types
2023-09-24 14:55:46 +02:00
meatbag-18a
ad64e33aad
Fix export-lora.cpp "not enough space in the context's memory pool" ( #1 )
...
* Fix export-lora.cpp "not enough space in the context's memory pool"
Without this patch, export-lora would sometimes error with "not enough space in the context's memory pool (needed 656784, available 656800)".
* increase required context size by 5*GGML_MEM_ALIGN instead of plain 16
---------
Co-authored-by: xaedes <xaedes@gmail.com>
2023-09-24 14:48:19 +02:00
xaedes
2912f17010
improve handling of export-lora arguments
...
print errors and warnings when files could not be read or created
2023-09-24 14:42:52 +02:00
slaren
c091cdfb24
llama-bench : add README ( #3317 )
...
* llama-bench : add README
* minor edit
2023-09-23 21:48:24 +02:00
Cebtenzzre
51a7cf5c6e
examples : fix RoPE defaults to match PR #3240 ( #3315 )
2023-09-23 12:28:50 +03:00
Kevin Ji
bedb92b603
scripts : use /usr/bin/env in shebang ( #3313 )
2023-09-22 23:52:23 -04:00
xaedes
da05205af6
cancel optimization when specified number of epochs is completed
2023-09-22 21:00:46 +02:00
xaedes
9145c87acc
update finetune README.md
2023-09-22 20:54:00 +02:00
xaedes
758c46cf7d
add export-lora build dependency to llama
...
because it depends on common, which depends on llama
2023-09-22 20:20:13 +02:00
xaedes
904c19bebd
remove unused struct tensor_info from export-lora
2023-09-22 20:19:12 +02:00
xaedes
d38260be0f
add export-lora build in Makefile
2023-09-22 20:04:07 +02:00
xaedes
b91e3dd2ee
remove trailing whitespace
2023-09-22 20:01:53 +02:00
xaedes
0ede0f4434
add export-lora program
2023-09-22 20:00:59 +02:00
Lee Drake
bc9d3e3971
Update README.md ( #3289 )
...
* Update README.md
* Update README.md
Co-authored-by: slaren <slarengh@gmail.com>
---------
Co-authored-by: slaren <slarengh@gmail.com>
2023-09-21 21:00:24 +02:00
shibe2
36b904e200
ggml-opencl.cpp: Make private functions static ( #3300 )
2023-09-21 14:10:26 -04:00
Edward Taylor
324f3403d5
zig : fix for updated c lib ( #3259 )
2023-09-21 12:08:20 +03:00
yuiseki
f56c418ab0
embedding : update README.md ( #3224 )
2023-09-21 11:57:40 +03:00
Johannes Gäßler
8185710a80
CUDA: use only 1 thread if fully offloaded ( #2915 )
2023-09-21 11:43:53 +03:00
Georgi Gerganov
7eb41179ed
readme : update hot topics
2023-09-20 20:48:22 +03:00
Cebtenzzre
a5661d7e71
llama : allow gguf RoPE keys to be overridden with defaults ( #3240 )
2023-09-20 12:12:47 -04:00
Cebtenzzre
65c2c1c5ab
benchmark-matmult : do not use integer abs() on a float ( #3277 )
2023-09-20 12:06:08 -04:00
kang
80834daecf
flake : Restore default package's buildInputs ( #3262 )
2023-09-20 15:48:22 +02:00
Alon
a40f2b656f
CI: FreeBSD fix ( #3258 )
...
* - freebsd ci: use qemu
2023-09-20 14:06:36 +02:00
Georgi Gerganov
d119c04c15
examples : fix benchmark-matmult ( #1554 )
...
The precision for Q4_0 has degraded since #1508
2023-09-20 10:02:39 +03:00
Cebtenzzre
8781013ef6
make : restore build-info.h dependency for several targets ( #3205 )
2023-09-18 10:03:53 -04:00
Erik Scholz
7ddf185537
ci : switch cudatoolkit install on windows to networked ( #3236 )
2023-09-18 02:21:47 +02:00
Johannes Gäßler
ee66942d7e
CUDA: fix peer access logic ( #3231 )
2023-09-17 23:35:20 +02:00
xaedes
5ce74ee461
initialize opt->loss_after with zero
2023-09-17 19:42:19 +02:00
xaedes
3b9d97484c
remove unused options and equalize train-text-from-scratch with finetune
2023-09-17 18:01:16 +02:00
xaedes
c993246bfd
train-text-from-scratch: automatically allocate compute memory
2023-09-17 17:52:22 +02:00
xaedes
f9b5d9b760
train-text-from-scratch: automatically allocate input tensors
2023-09-17 17:45:53 +02:00
xaedes
db38d2bce4
train-text-from-scratch: automatically allocate opt context
2023-09-17 17:33:11 +02:00
xaedes
9e10fa977e
train-text-from-scratch: automatically allocate model tensors, remove option '--mem-model N'
2023-09-17 17:14:11 +02:00
xaedes
dd94ce4ec0
remove now unnecessary llama API functions to get model params that where added by this PR
2023-09-17 16:51:15 +02:00
xaedes
934ad8d35d
move some params from lora hparams into model hparams and load model params from gguf
...
this equalizes the model definition in finetune and text-from-scratch and removes the need for additional llama api functions to get model parameters
2023-09-17 16:51:15 +02:00
