vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ggml : add RPC backend (#6829)

Browse source 
* ggml : add RPC backend

The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).

* set TCP_NODELAY

* add CI workflows

* Address review comments

* fix warning

* implement llama_max_devices() for RPC

* Address review comments

* Address review comments

* wrap sockfd into a struct

* implement get_alignment and get_max_size

* add get_device_memory

* fix warning

* win32 support

* add README

* readme : trim trailing whitespace

* Address review comments

* win32 fix

* Address review comments

* fix compile warnings on macos
This commit is contained in:
Radoslav Gerganov 2024-05-14 14:27:19 +03:00 • committed by GitHub
parent 541600201e
commit 5e31828d3e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 1395 additions and 98 deletions
Download patch file Download diff file Expand all files Collapse all files

2
examples/rpc/CMakeLists.txt Normal file
View file

@ -0,0 +1,2 @@
add_executable(rpc-server rpc-server.cpp)
target_link_libraries(rpc-server PRIVATE ggml llama)