vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Eliminate tokenizes post, add option "special" to tokenize

Browse source 
Eliminate tab compression from modified files.
This commit is contained in:
Troy Beukema 2023-10-23 11:48:37 -04:00
parent 5872e4f4da
commit 4150e74d04
2 changed files with 1762 additions and 1776 deletions
Download patch file Download diff file Expand all files Collapse all files

10
examples/server/README.md
View file

@ -222,18 +222,12 @@ node index.js
*Options:* *Options:*
`special`: Boolean indicating if special tokens should be tokenized (default false)
`content`: Set the text to tokenize. `content`: Set the text to tokenize.
Note that the special `BOS` token is not added in front of the text and also a space character is not inserted automatically as it is for `/completion`. Note that the special `BOS` token is not added in front of the text and also a space character is not inserted automatically as it is for `/completion`.
- **POST** `/tokenizes`: Tokenize a given text with special tokens.
*Options:*
`content`: Set the text to tokenize with special tokens.
Note that the special `BOS` token is not added in front of the text and also a space character is not inserted automatically as it is for `/completion`.
- **POST** `/detokenize`: Convert tokens to text. - **POST** `/detokenize`: Convert tokens to text.
*Options:* *Options:*

3528
examples/server/server.cpp
View file

File diff suppressed because it is too large Load diff