3 commits

Author	SHA1	Message	Date
Georgi Gerganov	43e12ce8e5	llama : use new pre-tokenizer type	2024-04-26 20:08:57 +03:00
Georgi Gerganov	e1b2bf783e	tests : add sample usage	2024-04-26 13:43:54 +03:00
Georgi Gerganov	aeafb43ed7	tests : remove and rename tokenizer test scripts	2024-04-26 13:39:03 +03:00

Renamed from tests/test-tokenizer-0-llama.py (Browse further)