vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Remove trailing whitespace in test-tokenizer-0-falcon.cpp

Browse source
This commit is contained in:
bobqianic 2024-02-12 11:05:47 +00:00 • committed by GitHub
parent 0da1e9c6c9
commit 0c612e5e04
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
tests/test-tokenizer-0-falcon.cpp
View file

@ -38,9 +38,9 @@ static const std::map<std::string, std::vector<llama_token>> & k_tests() {
{ " Hello\n Hello" , { 466, 23090, 742, 23090, }, }, { " Hello\n Hello" , { 466, 23090, 742, 23090, }, },
// { "\n =" , { 1212, 40, }, }, // { "\n =" , { 1212, 40, }, },
// This test has been disabled. Please re-enable it once the following issue has been resolved. // This test has been disabled. Please re-enable it once the following issue has been resolved.
// Falcon requires extra steps during the pre-tokenization phase, as detailed in the pull request at https://github.com/ggerganov/llama.cpp/pull/5446. // Falcon requires extra steps during the pre-tokenization phase, as detailed in the pull request at https://github.com/ggerganov/llama.cpp/pull/5446.
// For more information on pre-tokenizers, you can refer to the Hugging Face documentation: // For more information on pre-tokenizers, you can refer to the Hugging Face documentation:
// https://huggingface.co/docs/tokenizers/en/api/pre-tokenizers#tokenizers.pre_tokenizers. // https://huggingface.co/docs/tokenizers/en/api/pre-tokenizers#tokenizers.pre_tokenizers.
// It's crucial that we address this issue ASAP to avoid any unforeseen outcomes. // It's crucial that we address this issue ASAP to avoid any unforeseen outcomes.
{ "' era" , { 18, 4932, }, }, { "' era" , { 18, 4932, }, },