vbatts/llama.cpp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

tests : add sample usage

Browse source
This commit is contained in:
Georgi Gerganov 2024-04-26 13:43:54 +03:00
parent aeafb43ed7
commit e1b2bf783e
No known key found for this signature in database
GPG key ID: 449E073F9DC10735
2 changed files with 14 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
tests/test-tokenizer-0-bpe.py
View file

@ -1,4 +1,11 @@
# tests with BPE tokenizer # tests with BPE tokenizer
#
# sample usage:
#
# python3 tests/test-tokenizer-0-bpe.py ~/Data/huggingface/Meta-Llama-3-8B-Instruct/
# python3 tests/test-tokenizer-0-bpe.py ~/Data/huggingface/falcon-7b/
# python3 tests/test-tokenizer-0-bpe.py ~/Data/huggingface/deepseek-coder-6.7b-instruct/
#
import argparse import argparse

7
tests/test-tokenizer-0-spm.py
View file

@ -1,4 +1,11 @@
# tests with SPM tokenizer # tests with SPM tokenizer
#
# sample usage:
#
# python3 tests/test-tokenizer-0-spm.py ~/Data/huggingface/Llama-2-7b-hf/
# python3 tests/test-tokenizer-0-spm.py ~/Data/huggingface/CodeLlama-34b-Instruct-hf/
#
import argparse import argparse