vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add example of token splitting

Browse source
This commit is contained in:
Mathijs Henquet 2024-08-22 00:41:44 +02:00
parent b11e63ce43
commit 42fb6707e8
1 changed files with 10 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

10
examples/server/README.md
View file

@ -529,6 +529,16 @@ If `with_pieces` is `true`:
} }
``` ```
With input 'á' (utf8 hex: C3 A1) on tinyllama/stories260k
```json
{
"tokens": [
{"id": 198, "piece": [195]}, // hex C3
{"id": 164, "piece": [161]} // hex A1
]
}
```
### POST `/detokenize`: Convert tokens to text ### POST `/detokenize`: Convert tokens to text
*Options:* *Options:*