vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

server : Add tokenize with pieces tests to server.feature

Browse source
This commit is contained in:
Mathijs Henquet 2024-08-22 00:04:21 +02:00
parent a2d4d1913c
commit 198daa4e34
2 changed files with 10 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

8
examples/server/tests/features/server.feature
View file

@ -105,6 +105,14 @@ Feature: llama.cpp server
Given first token is removed Given first token is removed
Then tokens can be detokenized Then tokens can be detokenized
Scenario: Tokenize with pieces
When tokenizing with pieces:
"""
What is the capital of Germany?
媽
"""
Then tokens are given with pieces
Scenario: Models available Scenario: Models available
Given available models Given available models
Then 1 models are supported Then 1 models are supported

2
examples/server/tests/features/steps/steps.py
View file

@ -702,7 +702,7 @@ async def step_tokenize_with_pieces(context):
context.tokens_with_pieces = tokenize_json["tokens"] context.tokens_with_pieces = tokenize_json["tokens"]
@step("tokens with pieces are complete") @step("tokens are given with pieces")
@async_run_until_complete @async_run_until_complete
async def step_tokenize_with_pieces(context): async def step_tokenize_with_pieces(context):
# Verify that the response contains both token IDs and pieces # Verify that the response contains both token IDs and pieces