convert : add t5 tokenizer tests
This commit is contained in:
parent
6dc9eb4040
commit
9eb5d5617d
27 changed files with 47 additions and 4 deletions
|
@ -31,6 +31,7 @@
|
|||
1027
|
||||
1005 3690
|
||||
7592 1010 1061 1005 2035 999 2129 2024 2017 100 1029 1855 100 100 6207 100 100 14677 23632 22203 1811 1995
|
||||
999 999 999 999 999 999
|
||||
1017
|
||||
3943
|
||||
21211
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue