Find a file
2023-04-18 21:39:05 +08:00
.github Merged upstream, fixed OSX compile errors, integrated noavx2 build into main 2023-04-12 18:08:55 +08:00
CL integrated optional (experimentl) CLBlast support 2023-04-11 23:33:44 +08:00
examples Add LoRA support (#820) 2023-04-17 17:28:55 +02:00
lib integrated optional (experimentl) CLBlast support 2023-04-11 23:33:44 +08:00
media media : add logos and banners 2023-04-05 18:58:31 +03:00
models Make loading weights 10-100x faster 2023-03-30 12:28:25 -07:00
otherarch rwkv is done 2023-04-18 20:55:01 +08:00
prompts Revert "main : alternative instruct mode (Vicuna support, etc.) (#863)" (#982) 2023-04-14 22:58:43 +03:00
spm-headers deploy : add a Package.swift for SwiftPM support (#393) 2023-03-28 19:39:01 +03:00
tests llama : well-defined static initialization of complex objects (#927) 2023-04-17 17:41:53 +03:00
.ecrc Fix whitespace, add .editorconfig, add GitHub workflow (#883) 2023-04-11 19:45:44 +00:00
.editorconfig do not force the prompt file to end with a new line (#908) 2023-04-13 11:33:16 +02:00
.gitignore Merge branch 'master' into concedo 2023-04-14 21:40:33 +08:00
build.zig zig : update build.zig (#872) 2023-04-13 16:43:22 +03:00
cblas.h integrated optional (experimentl) CLBlast support 2023-04-11 23:33:44 +08:00
clblast.dll integrated optional (experimentl) CLBlast support 2023-04-11 23:33:44 +08:00
clblast_c.h integrated optional (experimentl) CLBlast support 2023-04-11 23:33:44 +08:00
convert-lora-to-ggml.py Add LoRA support (#820) 2023-04-17 17:28:55 +02:00
convert-pth-to-ggml.py py : new conversion script (#545) 2023-04-14 10:03:03 +03:00
convert.py add 4_0 to default outfile namestr dict (#1031) 2023-04-17 20:26:23 +02:00
export_state_dict_checkpoint.py added HF converter base 2023-03-31 19:10:21 +08:00
expose.cpp updated kobold lite, work on rwkv, added exe path to model load params, added launch parameter 2023-04-18 17:36:44 +08:00
expose.h updated kobold lite, work on rwkv, added exe path to model load params, added launch parameter 2023-04-18 17:36:44 +08:00
ggml.c Merge branch 'master' into concedo_experimental 2023-04-18 17:38:10 +08:00
ggml.h Add LoRA support (#820) 2023-04-17 17:28:55 +02:00
ggml_blas_adapter.c converted the cl file to be a string literal instead 2023-04-16 15:57:30 +08:00
ggml_clblast_dequant.cl converted the cl file to be a string literal instead 2023-04-16 15:57:30 +08:00
gpttype_adapter.cpp rwkv is done 2023-04-18 20:55:01 +08:00
klite.embd rwkv is done 2023-04-18 20:55:01 +08:00
koboldcpp.py updated kobold lite, work on rwkv, added exe path to model load params, added launch parameter 2023-04-18 17:36:44 +08:00
libopenblas.dll integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
LICENSE.md update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
llama.cpp Merge branch 'master' into concedo_experimental 2023-04-18 17:38:10 +08:00
llama.h Add LoRA support (#820) 2023-04-17 17:28:55 +02:00
llama_adapter.cpp arranged files, updated kobold lite, modified makefile for extra link args on linux, started RWKV implementation 2023-04-17 17:31:45 +08:00
llama_util.h Merge branch 'master' into concedo_experimental 2023-04-18 17:38:10 +08:00
llamaextra.cpp clean and refactor handling of flags 2023-04-12 23:25:31 +08:00
llamaextra.h clean and refactor handling of flags 2023-04-12 23:25:31 +08:00
make_pyinstaller.bat Merge branch 'concedo' into concedo_experimental 2023-04-18 17:42:43 +08:00
Makefile rwkv is done 2023-04-18 20:55:01 +08:00
MIT_LICENSE_GGML_LLAMACPP_ONLY rebrand to koboldcpp 2023-04-03 10:35:18 +08:00
model_adapter.cpp arranged files, updated kobold lite, modified makefile for extra link args on linux, started RWKV implementation 2023-04-17 17:31:45 +08:00
model_adapter.h arranged files, updated kobold lite, modified makefile for extra link args on linux, started RWKV implementation 2023-04-17 17:31:45 +08:00
niko.ico rebrand to koboldcpp 2023-04-03 10:35:18 +08:00
openblas_config.h integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
Package.swift Add Accelerate/BLAS when using Swift (#765) 2023-04-05 06:44:24 -04:00
preview.png resize image 2023-03-22 16:21:40 +08:00
README.md update readme 2023-04-18 21:39:05 +08:00
requirements.txt py : bump sentencepiece to 0.1.98 to support Python 3.11 (#976) 2023-04-14 19:46:49 +00:00
rwkv_vocab.embd updated kobold lite, work on rwkv, added exe path to model load params, added launch parameter 2023-04-18 17:36:44 +08:00
SHA256SUMS Revert "Delete SHA256SUMS for now" (#429) 2023-03-23 15:15:48 +01:00

koboldcpp (formerly llamacpp-for-kobold)

A self contained distributable from Concedo that exposes llama.cpp function bindings, allowing it to be used via a simulated Kobold API endpoint.

What does it mean? You get llama.cpp with a fancy UI, persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer. In a tiny package around 10 MB in size, excluding model weights.

Preview

Highlights

  • Now has experimental CLBlast support.
  • Now supports RWKV models WITHOUT pytorch or tokenizers! Yep, just GGML!

Usage

  • Download the latest release here or clone the repo.
  • Windows binaries are provided in the form of koboldcpp.exe, which is a pyinstaller wrapper for a few .dll files and koboldcpp.py. If you feel concerned, you may prefer to rebuild it yourself with the provided makefiles and scripts.
  • Weights are not included, you can use the official llama.cpp quantize.exe to generate them from your official weight files (or download them from other places).
  • To run, execute koboldcpp.exe or drag and drop your quantized ggml_model.bin file onto the .exe, and then connect with Kobold or Kobold Lite.
  • By default, you can connect to http://localhost:5001
  • You can also run it using the command line koboldcpp.exe [ggml_model.bin] [port]. For info, please check koboldcpp.exe --help
  • If you are having crashes or issues with OpenBLAS, please try the --noblas flag.

Compiling at Windows

  • If you want to compile your binaries from source at Windows, the easiest way is:
    • Use the latest release of w64devkit (https://github.com/skeeto/w64devkit). Be sure to use the "vanilla one", not i686 or other different stuff. If you try they will conflit with the precompiled libs!
    • Make sure you are using the w64devkit integrated terminal, then run 'make' at the KoboldCpp source folder. This will create the .dll files.
    • If you want to generate the .exe file, make sure you have the python module PyInstaller installed with pip ('pip install PyInstaller').
    • Run the script make_pyinstaller.bat at a regular terminal (or Windows Explorer).
    • The koboldcpp.exe file will be at your dist folder.

OSX and Linux

  • You will have to compile your binaries from source. A makefile is provided, simply run make
  • If you want you can also link your own install of OpenBLAS manually with make LLAMA_OPENBLAS=1
  • Alternatively, if you want you can also link your own install of CLBlast manually with make LLAMA_CLBLAST=1, for this you will need to obtain and link OpenCL and CLBlast libraries.
    • For Arch Linux: Install cblas and openblas. In the makefile, find the ifdef LLAMA_OPENBLAS conditional and add -lcblas to LDFLAGS.
    • For Debian: Install libclblast-dev and libopenblas-dev.
  • After all binaries are built, you can run the python script with the command koboldcpp.py [ggml_model.bin] [port]

Considerations

  • ZERO or MINIMAL changes as possible to parent repo files - do not move their function declarations elsewhere! We want to be able to update the repo and pull any changes automatically.
  • No dynamic memory allocation! Setup structs with FIXED (known) shapes and sizes for ALL output fields. Python will ALWAYS provide the memory, we just write to it.
  • For Windows: No installation, single file executable, (It Just Works)
  • Since v1.0.6, requires libopenblas, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without BLAS.
  • Since v1.15, requires CLBlast if enabled, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without CLBlast.
  • I plan to keep backwards compatibility with ALL past llama.cpp AND alpaca.cpp models. But you are also encouraged to reconvert/update your models if possible for best results.

License

  • The original GGML library and llama.cpp by ggerganov are licensed under the MIT License
  • However, Kobold Lite is licensed under the AGPL v3.0 License
  • The other files are also under the AGPL v3.0 License unless otherwise stated

Notes

  • Generation delay scales linearly with original prompt length. If OpenBLAS is enabled then prompt ingestion becomes about 2-3x faster. This is automatic on windows, but will require linking on OSX and Linux.
  • I have heard of someone claiming a false AV positive report. The exe is a simple pyinstaller bundle that includes the necessary python scripts and dlls to run. If this still concerns you, you might wish to rebuild everything from source code using the makefile, and you can rebuild the exe yourself with pyinstaller by using make_pyinstaller.bat
  • Supported GGML models:
    • LLAMA (All versions including ggml, ggmf, ggjt, gpt4all)
    • GPT-2 (All versions, including legacy f16, newer format + quanitzed, cerebras)
    • GPT-J (All versions including legacy f16, newer format + quantized, pyg.cpp, new pygmalion, janeway etc.),
    • RWKV (f16 GGMF format)
    • Basically every single current and historical GGML format that has ever existed should be supported, except for bloomz.cpp due to lack of demand.