5 commits

Author	SHA1	Message	Date
Georgi Gerganov	91eaa414bf	unicode : support \p{N}, \p{L} and \p{P} natively	2024-04-27 17:48:38 +03:00
Georgi Gerganov	4434c9d6c2	minor	2024-04-27 11:33:16 +03:00
Georgi Gerganov	43e12ce8e5	llama : use new pre-tokenizer type	2024-04-26 20:08:57 +03:00
Georgi Gerganov	e1b2bf783e	tests : add sample usage	2024-04-26 13:43:54 +03:00
Georgi Gerganov	aeafb43ed7	tests : remove and rename tokenizer test scripts	2024-04-26 13:39:03 +03:00

Renamed from tests/test-tokenizer-0-falcon.py (Browse further)