convert : add t5 tokenizer tests

This commit is contained in:
Georgi Gerganov 2024-07-02 10:39:49 +03:00
parent 6dc9eb4040
commit 9eb5d5617d
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
27 changed files with 47 additions and 4 deletions

View file

@ -31,6 +31,7 @@
1027
1005 3690
7592 1010 1061 1005 2035 999 2129 2024 2017 100 1029 1855 100 100 6207 100 100 14677 23632 22203 1811 1995
999 999 999 999 999 999
1017
3943
21211