convert : add t5 tokenizer tests

This commit is contained in:
Georgi Gerganov 2024-07-02 10:39:49 +03:00
parent 6dc9eb4040
commit 9eb5d5617d
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
27 changed files with 47 additions and 4 deletions

View file

@ -31,6 +31,7 @@
185 403
6 2906
17464 11 320 6 436 0 1724 418 340 33701 210 3025 19017 612 9407 2681 16 18 16 19 16 20 16 1398 68940 239
15278 3033
18
18 18
18 18 18