CUDA full GPU acceleration, KV cache in VRAM (#1827)

Browse source

* Fixed CUDA RoPE

* ggml_cuda_mul_mat_vec_p021

* ggml_cuda_scale

* ggml_cuda_diag_mask_inf

* ggml_is_permuted

* ggml_cuda_cpy

* flatten rows for ggml_cuda_op

* Added a --low-vram option

* Fixed Windows performance

* Fixed LLAMA_CUDA_DMMV_Y > 1 for WizardLM

...

This commit is contained in:

Johannes Gäßler 2023-06-14 19:47:19 +02:00 • committed by GitHub

parent 9254920265

commit 254a7a7a5f

No known key found for this signature in database

GPG key ID: 4AEE18F83AFDEB23

11 changed files with 853 additions and 149 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

797

ggml-cuda.cu

View file

File diff suppressed because it is too large Load diff