Add an API example using server.cpp similar to OAI. (#2009)

* add api_like_OAI.py
* add evaluated token count to server
* add /v1/ endpoints binding
This commit is contained in:
jwj7140 2023-07-05 03:06:12 +09:00 committed by GitHub
parent 7ee76e45af
commit f257fd2550
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 244 additions and 5 deletions

View file

@ -190,3 +190,19 @@ Run with bash:
```sh
bash chat.sh
```
### API like OAI
API example using Python Flask: [api_like_OAI.py](api_like_OAI.py)
This example must be used with server.cpp
```sh
python api_like_OAI.py
```
After running the API server, you can use it in Python by setting the API base URL.
```python
openai.api_base = "http://<Your api-server IP>:port"
```
Then you can utilize llama.cpp as an OpenAI's **chat.completion** or **text_completion** API