4 commits

Author	SHA1	Message	Date
Jared Van Bortel	87a6088ffe	rename unicodedata.{cpp,h} to unicode-data.{cpp,h}	2024-03-26 10:52:33 -04:00
Jared Van Bortel	b460e7f5b4	wpm : portable unicode tolower	2024-03-25 16:12:33 -04:00
Jared Van Bortel	e5ddf2fcdd	llama : split unicodedata.cpp from unicode.cpp	2024-03-25 16:12:32 -04:00
Georgi Gerganov	83796e62bc	llama : refactor unicode stuff (#5992) ... * llama : refactor unicode stuff ggml-ci * unicode : names * make : fix c++ compiler * unicode : names * unicode : straighten tables * zig : fix build * unicode : put nfd normalization behind API ggml-ci * swift : fix build * unicode : add BOM * unicode : add <cstdint> ggml-ci * unicode : pass as cpts as const ref	2024-03-11 17:47:47 +02:00