convert : fix gemma v1 tokenizer convert (#8248)

ggml-ci
This commit is contained in:
Georgi Gerganov 2024-07-04 10:41:03 +03:00 committed by GitHub
parent f619024764
commit 20fc3804bf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 85 additions and 4 deletions

View file

@ -91,6 +91,10 @@ __ggml_vocab_test__
__ggml_vocab_test__
333333333
__ggml_vocab_test__
Cửa Việt
__ggml_vocab_test__
discards
__ggml_vocab_test__