convert : add t5 tokenizer tests

This commit is contained in:
Georgi Gerganov 2024-07-02 10:39:49 +03:00
parent 6dc9eb4040
commit 9eb5d5617d
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
27 changed files with 47 additions and 4 deletions

View file

@ -31,6 +31,7 @@
203 280
25 34666
8279 30 533 25 464 19 4971 884 844 18458 228 1018 4982 13368 2909 9513 17827 35 37 35 38 35 39 35 11873 47838
9163 3202
37
37 37
37 37 37