Find a file
Concedo 86c7457e24 Merge branch 'master' into concedo
# Conflicts:
#	.github/workflows/build.yml
#	CMakeLists.txt
#	Makefile
#	README.md
#	main.cpp
2023-03-22 22:31:45 +08:00
.github/ISSUE_TEMPLATE Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
examples fix typo in chatLLaMa (#368) 2023-03-21 22:52:27 +02:00
models Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
prompts Add "--instruct" argument for usage with Alpaca (#240) 2023-03-19 18:37:02 +02:00
tests Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
.gitignore Nix flake (#40) 2023-03-17 23:03:48 +01:00
alpaca.sh Fix convert script, warnings alpaca instructions, default params 2023-03-21 17:59:16 +02:00
chat.sh Add chat.sh script 2023-03-21 18:09:46 +02:00
convert-gptq-to-ggml.py Importer for GPTQ quantized LLaMA models (#301) 2023-03-21 18:42:25 +02:00
convert-pth-to-ggml.py Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
download-pth.py 🚀 Dockerize llamacpp (#132) 2023-03-17 10:47:06 +01:00
expose.cpp Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
extra.cpp Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
extra.h Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
ggml.c Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
ggml.h Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
klite.embd added embedded copy of kobold lite 2023-03-21 20:41:19 +08:00
LICENSE.md update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
llama.cpp Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
llama.h Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
llama_for_kobold.py dynamic max context size defaulting to 1024, also implemented the basic API as a fallback 2023-03-22 15:56:47 +08:00
llamacpp.dll Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
main.cpp When seed <= 0 - use the clock to generate one 2023-03-22 07:47:15 +02:00
main.exe Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
Makefile Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
MIT_LICENSE_GGML_LLAMACPP_ONLY update license, added backwards compatibility with both ggml model formats, fixed context length issues. 2023-03-20 23:43:35 +08:00
preview.png resize image 2023-03-22 16:21:40 +08:00
quantize.cpp Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00
quantize.exe Merge branch 'master' into concedo 2023-03-22 22:31:45 +08:00
quantize.py Improved quantize script (#222) 2023-03-19 20:38:44 +02:00
README.md Updated readme 2023-03-22 16:20:37 +08:00
SHA256SUMS Add SHA256SUMS file and instructions to README how to obtain and verify the downloads 2023-03-21 23:19:11 +01:00
utils.cpp When seed <= 0 - use the clock to generate one 2023-03-22 07:47:15 +02:00
utils.h Introduce C-style API (#370) 2023-03-22 07:32:36 +02:00

llama-for-kobold

A self contained distributable from Concedo that exposes llama.cpp function bindings, allowing it to be used via a simulated Kobold API endpoint.

Preview

Considerations

  • Don't want to use pybind11 due to dependencies on MSVCC
  • ZERO or MINIMAL changes as possible to main.cpp - do not move their function declarations elsewhere!
  • Leave main.cpp UNTOUCHED, We want to be able to update the repo and pull any changes automatically.
  • No dynamic memory allocation! Setup structs with FIXED (known) shapes and sizes for ALL output fields. Python will ALWAYS provide the memory, we just write to it.
  • No external libraries or dependencies. That means no Flask, Pybind and whatever. All You Need Is Python.

Usage

  • Windows binaries are provided in the form of llamacpp.dll but if you feel worried go ahead and rebuild it yourself.
  • Weights are not included, you can use the llama.cpp quantize.exe to generate them from your official weight files (or download them from...places).
  • To run, simply clone the repo and run llama_for_kobold.py [ggml_quant_model.bin] [port], and then connect with Kobold or Kobold Lite.
  • By default, you can connect to http://localhost:5001 (you can also use https://lite.koboldai.net/?local=1&port=5001).

License

  • The original GGML library and llama.cpp by ggerganov are licensed under the MIT License
  • However, Kobold Lite is licensed under the AGPL v3.0 License
  • The provided python ctypes bindings in llamacpp.dll are also under the AGPL v3.0 License

Notes

  • There is a fundamental flaw with llama.cpp, which causes generation delay to scale linearly with original prompt length. If you care, please contribute to this discussion which, if resolved, will actually make this viable.