convert : add t5 tokenizer tests
This commit is contained in:
parent
6dc9eb4040
commit
9eb5d5617d
27 changed files with 47 additions and 4 deletions
|
@ -31,6 +31,7 @@
|
|||
198 796
|
||||
6 6980
|
||||
15496 11 331 6 439 0 1374 389 345 30325 223 5633 22755 239 46349 111 28839 101 18040 32432 98 43291 1485 1415 24309 25465 171 121 252
|
||||
13896 3228
|
||||
18
|
||||
2091
|
||||
20370
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue