Find a file
2023-03-30 00:48:44 +08:00
.github/ISSUE_TEMPLATE CI: fix subdirectory path globbing (#546) 2023-03-28 22:43:25 +03:00
examples llama : fix linkage with mingw (#551) 2023-03-28 21:23:09 +03:00
models Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
prompts Add longer DAN prompt for testing big batch numbers 2023-03-25 16:49:09 +02:00
spm-headers deploy : add a Package.swift for SwiftPM support (#393) 2023-03-28 19:39:01 +03:00
tests tests : free llama context at the end of the test 2023-03-28 19:51:55 +03:00
.gitignore integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
cblas.h integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
convert-gptq-to-ggml.py Fix GPTQ converter (#423) 2023-03-23 22:18:13 +02:00
convert-pth-to-ggml.py py : removed unused model variable and verified that the code functions correctly with vocab_only setting. Also confirmed that the code works as expected after running with reduced memory usage due to deletion of no-longer-needed variable. (#547) 2023-03-28 20:02:34 +03:00
convert-unversioned-ggml-to-ggml.py py : add temporary script to convert old ggml files to newer version (#539) 2023-03-28 20:55:42 +03:00
convert_ggml_to_pth.py py : add capabiliy to convert from ggml back to torch or hf format for further consumption/training/finetuning (#403) 2023-03-28 20:51:29 +03:00
expose.cpp integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
extra.cpp Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
extra.h Merge branch 'master' into concedo 2023-03-26 14:52:08 +08:00
ggml.c Enable Fused-Multiply-Add (FMA) and F16C/CVT16 vector extensions on MSVC (#375) 2023-03-28 22:44:29 +03:00
ggml.h ggml : introduce structs for the q4 data blocks (#356) 2023-03-28 18:56:03 +03:00
klite.embd Merged PR with a few changes: 2023-03-29 20:38:57 +08:00
libopenblas.dll integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
libopenblas.lib integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
LICENSE.md update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
llama.cpp Merge branch 'master' into concedo 2023-03-29 21:08:03 +08:00
llama.h llama : fix linkage with mingw (#551) 2023-03-28 21:23:09 +03:00
llamacpp.dll integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
llamacpp_blas.dll integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
llamacpp_for_kobold.py renamed main python script 2023-03-30 00:48:44 +08:00
main.exe Merge branch 'master' into concedo 2023-03-29 21:08:03 +08:00
make_pyinstaller.bat renamed main python script 2023-03-30 00:48:44 +08:00
Makefile integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
MIT_LICENSE_GGML_LLAMACPP_ONLY update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
niko.ico added a GUI for selection of models if none was passed in through command line. 2023-03-24 22:03:57 +08:00
openblas_config.h integrated libopenblas for greatly accelerated prompt processing. Windows binaries are included - feel free to build your own or to build for other platforms, but that is beyond the scope of this repo. Will fall back to non-blas if libopenblas is removed. 2023-03-30 00:43:52 +08:00
Package.swift deploy : add a Package.swift for SwiftPM support (#393) 2023-03-28 19:39:01 +03:00
preview.png resize image 2023-03-22 16:21:40 +08:00
quantize.exe Merge branch 'master' into concedo 2023-03-29 21:08:03 +08:00
quantize.py Check the existence of f16_model_path_base in quantize.py (#574) 2023-03-28 18:06:28 +03:00
README.md renamed main python script 2023-03-30 00:48:44 +08:00
SHA256SUMS Revert "Delete SHA256SUMS for now" (#429) 2023-03-23 15:15:48 +01:00

llamacpp-for-kobold

A self contained distributable from Concedo that exposes llama.cpp function bindings, allowing it to be used via a simulated Kobold API endpoint.

What does it mean? You get llama.cpp with a fancy UI, persistent stories, editing tools, save formats, memory, world info, author's note, characters, scenarios and everything Kobold and Kobold Lite have to offer. In a tiny package under 1 MB in size, excluding model weights.

Preview

Usage

  • Download the latest release here or clone the repo.
  • Windows binaries are provided in the form of llamacpp-for-kobold.exe, which is a pyinstaller wrapper for llamacpp.dll and llamacpp_for_kobold.py. If you feel concerned, you may prefer to rebuild it yourself with the provided makefiles and scripts.
  • Weights are not included, you can use the quantize.exe to generate them from your official weight files (or download them from other places).
  • To run, execute llamacpp-for-kobold.exe or drag and drop your quantized ggml_model.bin file onto the .exe, and then connect with Kobold or Kobold Lite.
  • By default, you can connect to http://localhost:5001
  • You can also run it using the command line llamacpp-for-kobold.exe [ggml_model.bin] [port]. For info, please check llamacpp-for-kobold.exe --help
  • If you are having crashes or issues with OpenBLAS, please try the --noblas flag.

OSX and Linux

  • You will have to compile your binaries from source. A makefile is provided, simply run make
  • After all binaries are built, you can run the python script with the command llamacpp_for_kobold.py [ggml_model.bin] [port]

Considerations

  • Don't want to use pybind11 due to dependencies on MSVCC
  • ZERO or MINIMAL changes as possible to main.cpp - do not move their function declarations elsewhere!
  • Leave main.cpp UNTOUCHED, We want to be able to update the repo and pull any changes automatically.
  • No dynamic memory allocation! Setup structs with FIXED (known) shapes and sizes for ALL output fields. Python will ALWAYS provide the memory, we just write to it.
  • No external libraries or dependencies. That means no Flask, Pybind and whatever. All You Need Is Python.
  • Since v1.0.6, requires libopenblas, the prebuilt windows binaries are included in this repo. If not found, it will fall back to a mode without BLAS.

License

  • The original GGML library and llama.cpp by ggerganov are licensed under the MIT License
  • However, Kobold Lite is licensed under the AGPL v3.0 License
  • The provided python ctypes bindings in llamacpp.dll are also under the AGPL v3.0 License

Notes

  • There is a fundamental flaw with llama.cpp, which causes generation delay to scale linearly with original prompt length. If you care, please contribute to this discussion which, if resolved, will actually make this viable.