convert : add t5 tokenizer tests

This commit is contained in:
Georgi Gerganov 2024-07-02 10:39:49 +03:00
parent 6dc9eb4040
commit 9eb5d5617d
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
27 changed files with 47 additions and 4 deletions

View file

@ -31,6 +31,7 @@
198 796
6 6980
15496 11 331 6 439 0 1374 389 345 30325 223 5633 22755 239 46349 111 28839 101 18040 32432 98 43291 1485 1415 24309 25465 171 121 252
13896 3228
18
2091
20370