convert : add t5 tokenizer tests
This commit is contained in:
parent
6dc9eb4040
commit
9eb5d5617d
27 changed files with 47 additions and 4 deletions
|
@ -31,6 +31,7 @@
|
|||
206 1857
|
||||
14 4515
|
||||
28339 19 1770 14 1954 8 4070 1955 1933 80503 231 5691 12081 13336 2648 29325 14315 24 26 24 27 24 28 24 5123 18372
|
||||
57178 10251
|
||||
26
|
||||
26 26
|
||||
26 26 26
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue