vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Use Unicode Escape Sequence to replace encoded characters

Browse source 
The use of special characters within source files can break compiling on some computers with different region and language settings. Using Unicode escape sequences should allow for the code to be compiled on all setups without needing to change your computers settings or switch regions.
This commit is contained in:
Tim Miller 2023-08-26 23:17:48 +09:00
parent 771551a793
commit 9ee138e62f
1 changed files with 4 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

8
llama.cpp
View file

@ -955,10 +955,10 @@ struct llama_vocab {
id linefeed_id = 13; id linefeed_id = 13;
int find_bpe_rank(std::string token_left, std::string token_right) const { int find_bpe_rank(std::string token_left, std::string token_right) const {
replace_all(token_left, " ", "Ġ"); replace_all(token_left, " ", "\u0120");
replace_all(token_left, "\n", "Ċ"); replace_all(token_left, "\n", "\u010A");
replace_all(token_right, " ", "Ġ"); replace_all(token_right, " ", "\u0120");
replace_all(token_right, "\n", "Ċ"); replace_all(token_right, "\n", "\u010A");
auto it = bpe_ranks.find(std::make_pair(token_left, token_right)); auto it = bpe_ranks.find(std::make_pair(token_left, token_right));
if (it == bpe_ranks.end()) { if (it == bpe_ranks.end()) {