vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: fix infinite recursion

Browse source
This commit is contained in:
Joan Martinez 2024-05-07 18:14:33 +02:00
parent d9e29036aa
commit 043f298775
2 changed files with 6 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

9
examples/server/tests/features/embeddings.feature
View file

@ -16,17 +16,10 @@ Feature: llama.cpp server
Then the server is starting Then the server is starting
Then the server is healthy Then the server is healthy
Scenario: Embedding
When embeddings are computed for:
"""
What is the capital of Bulgaria ?
"""
Then embeddings are generated
Scenario: Tokenize / Detokenize complex Scenario: Tokenize / Detokenize complex
When tokenizing: When tokenizing:
""" """
España is a èspciâl café über naïve résumé cañón élite cañas Barça 例子 東京 こんにちは 你好 中国 España is your's mine's l'heure èspciâl café über naïve résumé cañón élite cañas Barça 例子 東京 こんにちは 你好 中国
""" """
Then tokens can be detokenize and is equivalent False Then tokens can be detokenize and is equivalent False

6
unicode.cpp
View file

@ -496,7 +496,11 @@ std::vector<uint32_t> canonical_decomposition_cpts(std::vector<uint32_t> & cpts,
if (it.first != it.second) { if (it.first != it.second) {
uint offset = 0; uint offset = 0;
for (auto jt = it.first; jt != it.second; jt++) { for (auto jt = it.first; jt != it.second; jt++) {
cpts.emplace(cpts.begin() + i + offset, jt->second); if (offset == 0) {
cpts[i] = jt->second;
} else {
cpts.emplace(cpts.begin() + i + offset, jt->second);
}
offset++; offset++;
} }
const auto & inner_result = canonical_decomposition_cpts(cpts, i); const auto & inner_result = canonical_decomposition_cpts(cpts, i);