vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add more tokenizer tests (#3742)

Browse source 
* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test
This commit is contained in:
Galunid 2023-10-24 09:17:17 +02:00 • committed by GitHub
parent 469c9addef
commit daab3d7f45
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 16 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
models/ggml-vocab-baichuan.gguf Normal file
View file

Binary file not shown.

BIN
models/ggml-vocab-gpt-neox.gguf Normal file
View file

Binary file not shown.

BIN
models/ggml-vocab-refact.gguf Normal file
View file

Binary file not shown.

BIN
models/ggml-vocab-starcoder.gguf Normal file
View file

Binary file not shown.