vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fixed issue with gpt2 regex custom preprocessor

Browse source
This commit is contained in:
Kazim Abrar Mahi 2024-04-17 07:40:40 +06:00 • committed by Georgi Gerganov
parent 753580360b
commit 36d983262e
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
1 changed files with 1 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
unicode.cpp
View file

@ -357,6 +357,7 @@ static std::vector<size_t> unicode_gpt2_regex_preprocess(const std::wstring & wt
token += utf_char; token += utf_char;
} }
} }
start += offset;
} }
return bpe_offsets; return bpe_offsets;