vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Tokenizer SPM fixes for phi-3 and llama-spm (#7375)

Browse source 
* Update brute force test: special tokens
* Fix added tokens
  - Try to read 'added_tokens.json'.
  - Try to read 'tokenizer_config.json'.
  - Try to read 'tokenizer.json'.
* Fix special tokens rtrim

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* server : fix test regexes
This commit is contained in:
jaime-m-p 2024-05-20 20:15:57 +02:00 • committed by GitHub
parent fabf30b4c4
commit 917dc8cfa6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 98 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

4
examples/server/tests/features/slotsave.feature
View file

@ -26,7 +26,7 @@ Feature: llama.cpp server slot management
# Since we have cache, this should only process the last tokens
Given a user prompt "What is the capital of Germany?"
And a completion request with no api error
Then 24 tokens are predicted matching (Thank|special)
Then 24 tokens are predicted matching (Thank|special|Lily)
And 7 prompt tokens are processed
# Loading the original cache into slot 0,
# we should only be processing 1 prompt token and get the same output
@ -41,7 +41,7 @@ Feature: llama.cpp server slot management
Given a user prompt "What is the capital of Germany?"
And using slot id 1
And a completion request with no api error
Then 24 tokens are predicted matching (Thank|special)
Then 24 tokens are predicted matching (Thank|special|Lily)
And 1 prompt tokens are processed
Scenario: Erase Slot