vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: fix infinite recursion

Browse source
This commit is contained in:
Joan Martinez 2024-05-07 18:14:33 +02:00
parent d9e29036aa
commit 043f298775
2 changed files with 6 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

9
examples/server/tests/features/embeddings.feature
View file

@ -16,17 +16,10 @@ Feature: llama.cpp server
Then the server is starting
Then the server is healthy
Scenario: Embedding
When embeddings are computed for:
"""
What is the capital of Bulgaria ?
"""
Then embeddings are generated
Scenario: Tokenize / Detokenize complex
When tokenizing:
"""
España is a èspciâl café über naïve résumé cañón élite cañas Barça 例子 東京 こんにちは 你好 中国
España is your's mine's l'heure èspciâl café über naïve résumé cañón élite cañas Barça 例子 東京 こんにちは 你好 中国
"""
Then tokens can be detokenize and is equivalent False

6
unicode.cpp
View file

@ -496,7 +496,11 @@ std::vector<uint32_t> canonical_decomposition_cpts(std::vector<uint32_t> & cpts,
if (it.first != it.second) {
uint offset = 0;
for (auto jt = it.first; jt != it.second; jt++) {
cpts.emplace(cpts.begin() + i + offset, jt->second);
if (offset == 0) {
cpts[i] = jt->second;
} else {
cpts.emplace(cpts.begin() + i + offset, jt->second);
}
offset++;
}
const auto & inner_result = canonical_decomposition_cpts(cpts, i);