convert : add t5 tokenizer tests

This commit is contained in:
Georgi Gerganov 2024-07-02 10:39:49 +03:00
parent 6dc9eb4040
commit 9eb5d5617d
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
27 changed files with 47 additions and 4 deletions

View file

@ -31,6 +31,7 @@
185 405
6 2895
17535 11 320 6 435 0 1717 417 340 12394 233 210 3015 19100 608 9413 2668 16 18 16 19 16 20 16 1393 169 121 239
15330 3023
18
18 18
18 18 18