llama : refactor unicode stuff (#5992)

* llama : refactor unicode stuff ggml-ci * unicode : names * make : fix c++ compiler * unicode : names * unicode : straighten tables * zig : fix build * unicode : put nfd normalization behind API ggml-ci * swift : fix build * unicode : add BOM * unicode : add <cstdint> ggml-ci * unicode : pass as cpts as const ref
2024-03-11 17:47:47 +02:00 · 2024-03-11 17:47:47 +02:00 · 83796e62bc
commit 83796e62bc
parent 828defefb6
9 changed files with 1744 additions and 836 deletions
--- a/Package.swift
+++ b/Package.swift
@ -31,6 +31,7 @@ let package = Package(
            sources: [
                "ggml.c",
                "llama.cpp",
+                "unicode.cpp",
                "ggml-alloc.c",
                "ggml-backend.c",
                "ggml-quants.c",