Find a file
Concedo 1f6aa47b6e Merge branch 'master' into concedo
# Conflicts:
#	README.md
2023-04-11 16:53:41 +08:00
.github/ISSUE_TEMPLATE ci : re-enable AVX512 testing (Windows-MSVC) (#584) 2023-03-29 23:44:39 +03:00
examples Rewrite loading code to try to satisfy everyone: 2023-04-10 01:10:46 +02:00
media media : add logos and banners 2023-04-05 18:58:31 +03:00
models Make loading weights 10-100x faster 2023-03-30 12:28:25 -07:00
otherarch fixed a few OOM errors with larger contexts - I cannot figure out why they happen, so I am forced to increase the buffer size. 2023-04-11 00:14:57 +08:00
prompts add example of re-act pattern (#583) 2023-03-29 10:10:24 -05:00
spm-headers deploy : add a Package.swift for SwiftPM support (#393) 2023-03-28 19:39:01 +03:00
tests tests : free llama context at the end of the test 2023-03-28 19:51:55 +03:00
.gitignore Merge branch 'master' into concedo 2023-04-08 17:42:09 +08:00
build.zig zig : don't link examples/common.cpp for non-example (#814) 2023-04-07 19:05:29 +03:00
cblas.h integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
convert-ggml-to-pth.py py: huggingface -> Hugging Face (#686) 2023-04-01 18:38:18 +02:00
convert-gpt4all-to-ggml.py py : cleanup the code 2023-03-31 10:32:01 +02:00
convert-gptq-to-ggml.py py : cleanup the code 2023-03-31 10:32:01 +02:00
convert-pth-to-ggml.py py : cleanup the code 2023-03-31 10:32:01 +02:00
convert-unversioned-ggml-to-ggml.py py : cleanup the code 2023-03-31 10:32:01 +02:00
export_state_dict_checkpoint.py added HF converter base 2023-03-31 19:10:21 +08:00
expose.cpp new gpt2 format supported 2023-04-08 17:35:36 +08:00
expose.h Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
ggml.c ggml : fix WASM build 2023-04-10 23:20:01 +03:00
ggml.h ggml : add ggml_cont() + optimize ggml_cpy() for contiguous dst 2023-04-10 22:42:28 +03:00
gpttype_adapter.cpp fixed a few OOM errors with larger contexts - I cannot figure out why they happen, so I am forced to increase the buffer size. 2023-04-11 00:14:57 +08:00
klite.embd updated embedded kobold 2023-04-07 22:39:20 +08:00
koboldcpp.dll Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
koboldcpp.py Merge branch 'master' into concedo 2023-04-11 16:53:41 +08:00
koboldcpp_blas.dll Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
libopenblas.dll integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
libopenblas.lib integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
LICENSE.md update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
llama.cpp Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
llama.h Rewrite loading code to try to satisfy everyone: 2023-04-10 01:10:46 +02:00
llama_adapter.cpp Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
llama_internal.h Rewrite loading code to try to satisfy everyone: 2023-04-10 01:10:46 +02:00
llama_util.h Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
llamaextra.cpp Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
llamaextra.h Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
make_pyinstaller.bat rebrand to koboldcpp 2023-04-03 10:35:18 +08:00
Makefile Merged the upstream updates for model loading code, and ditched the legacy llama loaders since they were no longer needed. 2023-04-10 12:00:34 +08:00
migrate-ggml-2023-03-30-pr613.py py : cleanup the code 2023-03-31 10:32:01 +02:00
MIT_LICENSE_GGML_LLAMACPP_ONLY rebrand to koboldcpp 2023-04-03 10:35:18 +08:00
model_adapter.cpp added version label, improved file type checks 2023-04-10 01:03:09 +08:00
model_adapter.h new gpt2 format supported 2023-04-08 17:35:36 +08:00
niko.ico rebrand to koboldcpp 2023-04-03 10:35:18 +08:00
openblas_config.h integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
Package.swift Add Accelerate/BLAS when using Swift (#765) 2023-04-05 06:44:24 -04:00
preview.png resize image 2023-03-22 16:21:40 +08:00
README.md Add BAIR's Koala to supported models (#877) 2023-04-10 22:41:53 +02:00
SHA256SUMS Revert "Delete SHA256SUMS for now" (#429) 2023-03-23 15:15:48 +01:00

koboldcpp (formerly llamacpp-for-kobold)

A self contained distributable from Concedo that exposes llama.cpp function bindings, allowing it to be used via a simulated Kobold API endpoint.

What does it mean? You get llama.cpp with a fancy UI, persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer. In a tiny package around 10 MB in size, excluding model weights.

Preview

Highlights

  • This repo is somehow being flagged by Smartscreen in Microsoft Edge! I have no idea why, and it works fine in Chrome and Firefox. To be clear: Everything in this project is freely auditable and can be fully built from source code, any malware alerts are false positives! If you encounter such warnings, I'd appreciate it if you helped mark it as a false positive/safe to Microsoft.
  • Now has BACKWARDS COMPATIBILITY with ALL 3 versions of GGML LLAMA models, all ALPACA.CPP models, all versions of GPT4ALL.CPP models, and can run GGML older GPT-J.CPP models too. I also use a bunch of tricks to make subsequent prompt processing with shared ancestry much faster than the parent repo does.

Usage

  • Download the latest release here or clone the repo.
  • Windows binaries are provided in the form of koboldcpp.exe, which is a pyinstaller wrapper for koboldcpp.dll and koboldcpp.py. If you feel concerned, you may prefer to rebuild it yourself with the provided makefiles and scripts.
  • Weights are not included, you can use the quantize.exe to generate them from your official weight files (or download them from other places).
  • To run, execute koboldcpp.exe or drag and drop your quantized ggml_model.bin file onto the .exe, and then connect with Kobold or Kobold Lite.
  • By default, you can connect to http://localhost:5001
  • You can also run it using the command line koboldcpp.exe [ggml_model.bin] [port]. For info, please check koboldcpp.exe --help
  • If you are having crashes or issues with OpenBLAS, please try the --noblas flag.

OSX and Linux

  • You will have to compile your binaries from source. A makefile is provided, simply run make
  • If you want you can also link your own install of OpenBLAS manually with make LLAMA_OPENBLAS=1
    • For Arch Linux: Install cblas and openblas. In the makefile, find the ifdef LLAMA_OPENBLAS conditional and add -lcblas to LDFLAGS.
  • After all binaries are built, you can run the python script with the command koboldcpp.py [ggml_model.bin] [port]

Considerations

  • Don't want to use pybind11 due to dependencies on MSVCC
  • ZERO or MINIMAL changes as possible to parent repo files - do not move their function declarations elsewhere! We want to be able to update the repo and pull any changes automatically.
  • No dynamic memory allocation! Setup structs with FIXED (known) shapes and sizes for ALL output fields. Python will ALWAYS provide the memory, we just write to it.
  • No external libraries or dependencies. That means no Flask, Pybind and whatever. All You Need Is Python.
  • Since v1.0.6, requires libopenblas, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without BLAS.
  • I plan to keep backwards compatibility with ALL past llama.cpp AND alpaca.cpp models. But you are also encouraged to reconvert/update your models if possible for best results.

License

  • The original GGML library and llama.cpp by ggerganov are licensed under the MIT License
  • However, Kobold Lite is licensed under the AGPL v3.0 License
  • The provided python ctypes bindings in koboldcpp.dll are also under the AGPL v3.0 License

Notes

  • Generation delay scales linearly with original prompt length. See this discussion. If OpenBLAS is enabled then prompt ingestion becomes about 2-3x faster. This is automatic on windows, but will require linking on OSX and Linux.
  • I have heard of someone claiming a false AV positive report. The exe is a simple pyinstaller bundle that includes the necessary python scripts and dlls to run. If this still concerns you, you might wish to rebuild everything from source code using the makefile, and you can rebuild the exe yourself with pyinstaller by using make_pyinstaller.bat